Method and system for providing buffer management in a perormance enhancing proxy architecture

ABSTRACT

A network apparatus for providing performance enhancements of a communication network is disclosed. The network apparatus includes a plurality of communication interfaces that are configured to receive and to forward messages according to a prescribed protocol. The network apparatus also includes a plurality of modules configured to process the messages to effect performance enhancing functions. Further, the network apparatus includes a plurality of buffers that are configured to store the received messages and messages that are generated by one of the plurality of modules. A portion of the plurality of buffers is shared by the plurality of modules based upon execution of a particular one of the performance enhancing functions. Each of the plurality of buffers has a data structure that includes an expandable header to accommodate different message types. The present invention has particular applicability to a bandwidth constrained system, such as satellite network.

CROSS-REFERENCES TO RELATED APPLICATION

[0001] This application is related to and claims the benefit of priorityto: (i) U.S. Provisional Patent Application (Serial No. 60/220,026),filed Jul. 21, 2000, entitled “Performance Enhancing Proxy,” and (ii)U.S. Provisional Patent Application (Serial No. 60/225,630), filed Aug.15, 2000, entitled “Performance Enhancing Proxy”; all of which areincorporated herein by reference in their entirety.

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003] The present invention relates to a communication system, and ismore particularly related to a proxy architecture for improving networkperformance.

[0004] 2. Discussion of the Background

[0005] The entrenchment of data networking into the routines of modemsociety, as evidenced by the prevalence of the Internet, particularlythe World Wide Web, has placed ever-growing demands on service providersto continually improve network performance. To meet this challenge,service providers have invested heavily in upgrading their networks toincrease system capacity (i.e., bandwidth). In many circumstances, suchupgrades may not be feasible economically or the physical constraints ofthe communication system does not permit simply “upgrading.”Accordingly, service providers have also invested in developingtechniques to optimize the performance of their networks. Because muchof today's networks are either operating with or are required tointerface with the Transmission Control Protocol/Internet Protocol(TCP/IP) suite, attention has been focused on optimizing TCP/IP basednetworking operations.

[0006] As the networking standard for the global Internet, TCP/IP hasearned such acceptance among the industry because of its flexibility andrich heritage in the research community. The transmission controlprotocol (TCP) is the dominant protocol in use today on the Internet.TCP is carried by the Internet protocol (IP) and is used in a variety ofapplications including reliable file transfer and Internet web pageaccess applications. The four layers of the TCP/IP protocol suite areillustrated in FIG. 31. As illustrated, the link layer (or the networkinterface layer) 10 includes device drivers in the operating system andany corresponding network interface cards. Together, the device driverand the interface cards handle hardware details of physicallyinterfacing with any cable or whatever type of media that is being used.The network layer (also referred to as the Internet layer) 12 handlesthe movement of packets around the network. Routing of packets, forexample, takes place at the network layer 12. IP, Internet controlmessage protocol (ICMP), and Internet group management protocol (IGMP)may provide the network layer in the TCP/IP protocol suite. Thetransport layer 14 provides a flow of data between two hosts, for theapplication layer 16 above.

[0007] In the TCP/IP protocol suite, there are at least two differenttransport protocols, TCP and a user datagram protocol (UDP). TCP whichprovides a reliable flow of data between two hosts, is primarilyconcerned with dividing the data passed to it from the application layer16 into appropriately sized segments for the network layer 12 below,acknowledging received packets, setting timeouts to make certain theother end acknowledges packets that are sent, and so on. Because thisreliable flow of data is provided by the transport layer 14, theapplication layer 16 is isolated from these details. UDP, on the otherhand, provides a much simpler service to the application layer 16. UDPjust sends packets of data called datagrams from one host to another,with no guarantee that the datagrams will reach their destination. Anydesired reliability must be added by a higher layer, such as theapplication layer 16.

[0008] The application layer 16 handles the details of the particularapplication. There are many common TCP/IP applications that almost everyimplementation provides, including telnet for remote log-in, the filetransfer protocol (FTP), the simple mail transfer protocol (SMTP) orelectronic mail, the simple network management protocol (SNMP), thehypertext transfer protocol (HTTP), and many others.

[0009] As mentioned, TCP provides reliable, in-sequence delivery of databetween two IP hosts. The IP hosts set up a TCP connection, using aconventional TCP three-way handshake and then transfer data using awindow-based protocol with the successfully received data acknowledged.

[0010] To understand where optimizations may be made, it is instructiveto consider a typical TCP connection establishment. FIG. 32 illustratesan example of the conventional TCP three-way handshake between IP hosts20 and 22. First, the IP host 20 that wishes to initiate a transfer withIP host 22, sends a synchronize (SYN) signal to IP host 22. The IP host22 acknowledges the SYN signal from IP host 20 by sending a SYNacknowledgement (ACK). The third step of the conventional TCP three-wayhandshake is the issuance of an ACK signal from the IP host 20 to theother IP host 22. At this point, IP host 22 is ready to receive the datafrom IP host 20 (and vice versa). After all the data has been delivered,another handshake (similar to the handshake described to initiate theconnection) is used to close the TCP connection.

[0011] TCP was designed to be very flexible and to work over a widevariety of communication links, including both slow and fast links, highlatency links, and links with low and high error rates. However, whileTCP (and other high layer protocols) works with many different kinds oflinks, TCP performance, in particular, the throughput possible acrossthe TCP connection, is affected by the characteristics of the link inwhich it is used. There are many link layer design considerations thatshould be taken into account when designing a link layer service that isintended to support Internet protocols. However, not all characteristicscan be compensated for by choices in the link layer design. TCP has beendesigned to be very flexible with respect to the links which ittraverses. Such flexibility is achieved at the cost of sub-optimaloperation in a number of environments vis-{fraction (a)}-vis a tailoredprotocol. The tailored protocol which is usually proprietary in nature,may be more optimal, but greatly lacks flexibility in terms ofnetworking environments and interoperability.

[0012] An alternative to a tailored protocol is the use of performanceenhancing proxies (PEPs), to perform a general class of functions termed“TCP spoofing,” in order to improve TCP performance over impaired (i.e.,high latency or high error rate) links. TCP spoofing involves anintermediate network device (the performance enhancing proxy (PEP))intercepting and altering, through the addition and/or deletion of TCPsegments, the behavior of the TCP connection in an attempt to improveits performance.

[0013] Conventional TCP spoofing implementations include the localacknowledgement of TCP data segments in order to get the TCP data senderto send additional data sooner than it would have sent if spoofing werenot being performed, thus improving the throughput of the TCPconnection. Generally, conventional TCP spoofing implementations havefocused simply on increasing the throughput of TCP connections either byusing larger windows over the link or by using compression to reduce theamount of data which needs to be sent, or both.

[0014] Many TCP PEP implementations are based on TCP ACK manipulation.These may include TCP ACK spacing where ACKs which are bunched together,are spaced apart, local TCP ACKs, local TCP retransmissions, and TCP ACKfiltering and reconstruction. Other PEP mechanisms include tunneling,compression, and priority-based multiplexing.

[0015] Based on the foregoing, there is a clear need for improvedapproaches to optimizing network performance, while achieving networkflexibility. There is also a need to enhance network performance,without a costly infrastructure investment. There is also a need toemploy a network performance enhancing mechanism that complies withexisting standards to facilitate rapid deployment. There is a furtherneed to simplify the receiver design. Therefore, an approach foroptimizing network performance using a proxy architecture is highlydesirable.

SUMMARY OF THE INVENTION

[0016] The present invention addresses the above stated needs byproviding a network apparatus for providing performance enhancing proxy(PEP) functionalities. The network apparatus includes multiple buffersthat correspond to communication interfaces and that are utilized byperformance enhancing proxy (PEP) kernels. The buffers have a datastructure that provides an expandable field that adapts to differentmessage types.

[0017] According to one aspect of the invention, a network apparatus forproviding performance enhancements of a communication network isprovided. The network apparatus includes a plurality of communicationinterfaces that are configured to receive and to forward messagesaccording to a prescribed protocol. The network apparatus also includesa plurality of modules configured to process the messages to effectperformance enhancing functions. Further, the network apparatus includesa plurality of buffers that are configured to store the receivedmessages and messages that are generated by one of the plurality ofmodules. A portion of the plurality of buffers is shared by theplurality of modules based upon execution of a particular one of theperformance enhancing functions. Each of the plurality of buffers has adata structure that includes an expandable header to accommodatedifferent message types. This approach advantageously provides efficientmanagement of buffers within a network component.

[0018] According to another aspect of the invention, a method forproviding performance enhancements of a communication network isdisclosed. The method includes receiving messages according to aprescribed protocol, processing the messages to effect performanceenhancing functions via a plurality of modules, and storing the receivedmessages and messages that are generated by one of the plurality ofmodules in a plurality of buffers. A portion of the plurality of buffersis shared by the plurality of modules based upon execution of aparticular one of the performance enhancing functions, wherein each ofthe plurality of buffers has a data structure that includes anexpandable header to accommodate different message types. The abovearrangement advantageously improves system efficiency.

[0019] According to one aspect of the invention, a network apparatus forproviding performance enhancements of a communication network includesmeans for receiving messages according to a prescribed protocol, andmeans for processing the messages to effect performance enhancingfunctions. The received messages and messages that are generated byprocessing means are stored in a plurality of buffers. A portion of theplurality of buffers is shared by the processing means based uponexecution of a particular one of the performance enhancing functions.Each of the plurality of buffers has a data structure that includes anexpandable header to accommodate different message types. The abovearrangement advantageously provides efficient buffer management.

[0020] According to another aspect of the invention, a computer-readablemedium carrying one or more sequences of one or more instructions forproviding performance enhancements of a communication network isdisclosed. The one or more sequences of one or more instructions includeinstructions which, when executed by one or more processors, cause theone or more processors to perform the step receiving messages accordingto a prescribed protocol. Other steps include processing the messages toeffect performance enhancing functions via a plurality of modules, andstoring the received messages and messages that are generated by one ofthe plurality of modules in a plurality of buffers. A portion of theplurality of buffers is shared by the plurality of modules based uponexecution of a particular one of the performance enhancing functions.Each of the plurality of buffers has a data structure that includes anexpandable header to accommodate different message types. This approachadvantageously provides enhanced network performance.

[0021] In yet another aspect of the present invention, a memory forstoring information for providing performance enhancements of acommunication network is disclosed. The memory comprises a datastructure that includes a specific header field that stores platformspecific information. The data structure also includes a common headerfield that stores information known to the plurality of modules, and apayload field. Under this approach, efficient buffer management isachieved.

BRIEF DESCRIPTION OF THE DRAWINGS

[0022] A more complete appreciation of the invention and many of theattendant advantages thereof will be readily obtained as the samebecomes better understood by reference to the following detaileddescription when considered in connection with the accompanyingdrawings, wherein:

[0023]FIG. 1 is a diagram of a communication system in which theperformance enhancing proxy (PEP) of the present invention isimplemented;

[0024]FIG. 2 is a diagram of a PEP end point platform environment,according to an embodiment of the present invention;

[0025]FIG. 3 is a diagram of a TCP Spoofing Kernel (TSK) utilized in theenvironment of FIG. 2;

[0026]FIGS. 4A and 4B are flow diagrams of the connection establishmentwith three-way handshake spoofing and without three-way handshakespoofing, respectively;

[0027]FIG. 5 is a diagram of a PEP packet flow between two PEP endpoints, according to an embodiment of the present invention;

[0028]FIG. 6 is a diagram of an IP (Internet Protocol) packet flowthrough a PEP end point, in accordance with an embodiment of the presentinvention;

[0029]FIG. 7 is a diagram of PEP end point profiles utilized in theplatform of FIG. 2;

[0030]FIG. 8 is a diagram of the interfaces of a PEP end pointimplemented as an IP gateway, according to an embodiment of the presentinvention;

[0031]FIG. 9 is a diagram of the interfaces of a PEP end pointimplemented as a Multimedia Relay, according to an embodiment of thepresent invention;

[0032]FIG. 10 is a diagram of the interfaces of a PEP end pointimplemented as a Multimedia VSAT (Very Small Aperture Terminal),according to an embodiment of the present invention;

[0033]FIG. 11 is a diagram of the interfaces of a PEP end pointimplemented in an earth station, according to an embodiment of thepresent invention;

[0034]FIG. 12 is a diagram of the flow of TCP spoofing buffers through aPEP End Point, according to an embodiment of the present invention, inaccordance with an embodiment of the present invention;

[0035]FIG. 13 is a diagram of the buffer management for unspoofed TCPconnections and for non-TCP traffic, according to an embodiment of thepresent invention;

[0036]FIG. 14 is a diagram of a basic format of the buffers used toimplement the PEP functionality, according to an embodiment of thepresent invention;

[0037]FIG. 15 is a diagram of an IP packet that is used in the system ofFIG. 1;

[0038]FIG. 16 is a diagram of a format of the PEP common buffer header,according to an embodiment of the present invention;

[0039]FIG. 17 is a diagram of a received TCP data segment headeradjustment, according to an embodiment of the present invention;

[0040]FIG. 18 is a diagram of a received TCP data segment with a TCPconnection header, according to an embodiment of the present invention;

[0041]FIG. 19 is a diagram of a received TSK data message headeradjustment, according to an embodiment of the present invention;

[0042]FIG. 20 is a diagram of a received TSK data message headeradjustment with a TCP connection header, according to an embodiment ofthe present invention;

[0043]FIG. 21 is a diagram of a generated TCP segment, according to anembodiment of the present invention;

[0044]FIG. 22 is a diagram of a generated PBP segment, according to anembodiment of the present invention;

[0045]FIG. 23 is a diagram of a generated TSK message, according to anembodiment of the present invention;

[0046]FIG. 24 is a diagram showing reuse of a TCP segment buffer for aTSK message, according to an embodiment of the present invention;

[0047]FIG. 25 is a diagram of reuse of a TSK message buffer for a TCPsegment, according to an embodiment of the present invention;

[0048]FIG. 26 is a diagram of an exemplary kernel use of the ownerspecific “header”, according to an embodiment of the present invention;

[0049]FIG. 27 is a diagram of a process for inserting a PEP commonbuffer header into a small buffer, according to an embodiment of thepresent invention;

[0050]FIG. 28 is a diagram of a process for adding a PEP common bufferheader to small buffer, according to an embodiment of the presentinvention;

[0051]FIG. 29 is a diagram of a sliding window mechanism used in thesystem of FIG. 1, according to an embodiment of the present invention;

[0052]FIG. 30 is a diagram of a computer system that can perform PEPfunctions, in accordance with an embodiment of the present invention;

[0053]FIG. 31 is diagram of the protocol layers of the TCP/IP protocolsuite; and

[0054]FIG. 32 is diagram of a conventional TCP three-way handshakebetween IP hosts.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] In the following description, for the purpose of explanation,specific details are set forth in order to provide a thoroughunderstanding of the invention. However, it will be apparent that theinvention may be practiced without these specific details. In someinstances, well-known structures and devices are depicted in blockdiagram form in order to avoid unnecessarily obscuring the invention.

[0056] Although the present invention is discussed with respect to theInternet and the TCP/IP protocol suite, the present invention hasapplicability to other packet switched networks and equivalentprotocols.

[0057]FIG. 1 illustrates an exemplary network 100 in which theperformance enhancing proxy (PEP) of the present invention may beutilized. The network 100 in FIG. 1 includes one or more hosts 110connected to a network gateway 120 via TCP connections. The networkgateway 120 is connected to another network gateway 140 via a backboneconnection on a backbone link 130. As seen in FIG. 1, the backbone link130, in an exemplary embodiment, is shown as a satellite link that isestablished over a satellite 101; however, it is recognized by one ofordinary skill in the art that other network connections may beimplemented. For example, these network connections may be establishedover a wireless communications system, in general, (e.g., radionetworks, cellular networks, etc.) or a terrestrial communicationssystem. The network gateway 140 is further connected to a second groupof hosts 150, also via TCP connections. In the arrangement illustratedin FIG. 1, the network gateways 120, 140 facilitate communicationbetween the groups of hosts 110, 150.

[0058] The network gateways 120, 140 facilitate communication betweenthe two groups of hosts 110, 150 by performing a number of performanceenhancing functions. These network gateways 120, 140 may performselective TCP spoofing which allows flexible configuration of theparticular TCP connections that are to be spoofed. Additionally,gateways 120, 140 employ a TCP three-way handshake, in which the TCPconnections are terminated at each end of the backbone link 130. Localdata acknowledgements are utilized by the network gateways 120, 140,thereby permitting the TCP windows to increase at local speeds.

[0059] The network gateway 120, 140 further multiplexes multiple TCPconnections across a single backbone connection; this capability reducesthe amount of acknowledgement traffic associated with the data frommultiple TCP connections, as a single backbone connectionacknowledgement may be employed. The multiplexing function also providessupport for high throughput TCP connections, wherein the backboneconnection protocol is optimized for the particular backbone link thatis used. The network gateways 120, 140 also support data compressionover the backbone link 130 to reduce the amount of traffic to be sent,further leveraging the capabilities of the backbone connection. Further,the network gateways 120, 140 utilize data encryption in the datatransmission across the backbone link 130 to protect data privacy, andprovide prioritized access to backbone link 130 capacity on a per TCPconnection basis. Each of the network gateways 120, 140 may select aparticular path for the data associated with a connection to flow. Theabove capabilities of the network gateways 120, 140 are more fullydescribed below.

[0060]FIG. 2 illustrates a performance enhancing proxy (PEP) 200 asimplemented in a network gateway 120, 140, according to one embodimentof the present invention. In this embodiment, the PEP 200 has a platformenvironment 210 which includes the hardware and software operatingsystem. The PEP 200 also includes local area network (LAN) interfaces220 and wide area network (WAN) interfaces 230. In the example in FIG.1, the network gateway 120 may establish the TCP connections with the IPhosts 110, via a local LAN interface 220 and may establish the backboneconnection with the network gateway 140 via a WAN interface 230. The PEPplatform environment 210 may also include general functional modules:routing module 240, buffer management module 250, event managementmodule 260, and parameter management module 270. As illustrated in FIG.2, the network gateway also includes a TCP spoofing kernel (TSK) 280, abackbone protocol kernel (BPK) 282, a prioritization kernel (PK) 284,and a path selection kernel (PSK) 286. These four kernels essentiallymake up the functionality of the performance enhancing proxy 200.

[0061] The platform environment 210 performs a number of functions. Onesuch function is to shield the various PEP kernels 280, 282, 284, 286from implementation specific constraints. That is, the platformenvironment 210 performs functions that the various PEP kernels 280,282, 284, 286 cannot perform directly because the implementation of thefunction is platform specific. This arrangement has the advantageouseffect of hiding platform specific details from the PEP kernels 280,282, 284, 286, making the PEP kernels more portable. An example of aplatform specific function is the allocation of a buffer. In someplatforms, buffers are created as they are needed, while in otherplatforms, buffers are created at start-up and organized into linkedlists for later use. It is noted that platform specific functions arenot limited to functions generic to all of the kernels 280, 282, 284,286. A function specific to a particular kernel, for example, theallocation of a control block for TCP spoofing, may also be implementedin the platform environment to hide platform specific details from thekernel.

[0062] Additionally, the platform environment 210 may provide the taskcontext in which the PEP kernels 280, 282, 284, 286 run. In oneexemplary embodiment, all PEP kernels 280, 282, 284, 286 can run in thesame task context for efficiency. However, this is not required.

[0063] Furthermore, the platform environment 210 provides an interfacebetween the PEP functionality (embodied in kernels 280, 282, 284, 286)and the other functionality of the network gateway 120, 140. Forexample, the platform environment 210 may provide the interface betweenthe PEP functionality and the routing function 240, as seen in FIG. 2.It is noted that the platform specific functions illustrated in FIG. 2are examples and are not considered an exhaustive list. It is furthernoted that the PEP kernels shown touching each other (280, 282 and 284,286) in FIG. 2 may have a direct procedural interface to each other.Further, the kernels 280, 282, 284, 286 may include direct interfaces toimprove performance, as opposed to routing everything through theplatform environment 210 (as shown in FIG. 2).

[0064] In addition to the PEP kernels 280, 282, 284, and 286, the PEPend point platform 210 may utilize a data compression kernel (CK) 290and an encryption kernel (EK) 292. These kernels 280, 282, 284, 286,290, and 292, as described above, facilitate communication between thetwo groups of hosts 110, 150, by performing a variety of performanceenhancing functions, either singly or in combination. These performanceenhancing functions include selective TCP spoofing, three-way handshakespoofing, local data acknowledgement, TCP connection to backboneconnection multiplexing, data compression/encryption, prioritization,and path selection.

[0065] Selective TCP Spoofing is performed by the TSK 280 and includes aset of user configurable rules that are used to determine which TCPconnections should be spoofed. Selective TCP spoofing improvesperformance by not tying up TCP spoofing-related resources, such asbuffer space, control blocks, etc., for TCP connections for which theuser has determined that spoofing is not beneficial or required and bysupporting the use of tailored parameters for TCP connections that arespoofed.

[0066] In particular, the TSK 280 discriminates among the various TCPconnections based on the applications using them. That is, TSK 280discriminates among these TCP connections to determine which connectionshould be spoofed as well as the manner in which the connection isspoofed; e.g., whether to spoof the three-way handshake, the particulartimeout parameters for the spoofed connections, etc. TCP spoofing isthen performed only for those TCP connections that are associated withapplications for which high throughput or reduced connection startuplatency (or both) is required. As a result, the TSK 280 conserves TCPspoofing resources for only those TCP connections for which highthroughput or reduced connection startup latency (or both) is required.Further, the TSK 280 increases the total number of TCP connections whichcan be active before running out of TCP spoofing resources, since anyactive TCP connections which do not require high throughput, are notallocated resources.

[0067] One criterion for identifying TCP connections of applications forwhich TCP spoofing should and should not be performed is the TCP portnumber field contained in the TCP packets being sent. In general, uniqueport numbers are assigned to each type of application. Which TCP portnumbers should and should not be spoofed can be stored in the TSK 280.The TSK 280 is also re-configurable to allow a user or operator toreconfigure the TCP port numbers which should and should not be spoofed.The TSK 280 also permits a user or operator to control which TCPconnections are to be spoofed, based on other criteria. In general, adecision on whether to spoof a TCP connection may be based on any fieldwithin a TCP packet. The TSK 280 permits a user to specify which fieldsto examine and which values in these fields identify TCP connectionsthat should or should not be spoofed. Another example of a potential usefor this capability is for the user or operator to select the IP addressof the TCP packet in order to control for which users TCP spoofing isperformed. The TSK 280 also permits a user to look at multiple fields atthe same time. As a result, the TSK 280 permits a user or operator touse multiple criteria for selecting TCP connections to spoof. Forexample, by selecting both the IP address and the TCP port numberfields, the system operator can enable TCP spoofing for only specificapplications from specific users.

[0068] The user configurable rules may include five exemplary criteriawhich can be specified by the user or operator in producing a selectiveTCP spoofing rule: Destination IP address; Source IP address; TCP portnumbers (which may apply to both the TCP destination and source portnumbers); TCP options; and IP differentiated services (DS) field.However, as indicated above, other fields within the TCP packet may beused.

[0069] As discussed above, in addition to supporting selective TCPspoofing rules for each of these criterion, AND and OR combinationoperators can be used to link criteria together. For example, using theAND combination operator, a rule can be defined to disable TCP spoofingfor FTP data received from a specific host. Also, the order in which therules are specified may be significant. It is possible for a connectionto match the criteria of multiple rules. Therefore, the TSK 280 canapply rules in the order specified by the operator, taking the action ofthe first rule that matches. A default rule may also be set whichdefines the action to be taken for TCP connections which do not matchany of the defined rules. The set of rules selected by the operator maybe defined in a selective TCP spoofing selection profile.

[0070] As an example, assuming sufficient buffer space has beenallocated to spoof five TCP connections, if four low speed applications(i.e., applications which, by their nature, do not require high-speed)bring up connections along with one high-speed application, thehigh-speed connection has access to only 1/5 of the available spoofingbuffer space. Further, if five low speed connections are brought upbefore the high-speed connection, the high-speed connection cannot bespoofed at all. Using the TSK 280 selective spoofing mechanism, the lowspeed connections are not allocated any spoofing buffer space.Therefore, the high-speed connection always has access to all of thebuffer space, improving its performance with respect to animplementation without the selective TCP spoofing feature of the TSK280.

[0071] The TSK 280 also facilitates spoofing of the conventionalthree-way handshake. Three-Way Handshake Spoofing involves locallyresponding to a connection request to bring up a TCP connection inparallel with forwarding the connection requests across the backbonelink 130 (FIG. 1). This allows the originating IP host (for example,110) to reach the point of being able to send the data it must send atlocal speeds, i.e. speeds that are independent of the latency of thebackbone link 130. Three-way Handshake Spoofing allows the data that theIP host 110 needs to send to be sent to the destination IP host 150without waiting for the end-to-end establishment of the TCP connection.For backbone links 130 with high latency, this significantly reduces thetime it takes to bring up the TCP connection and, more importantly, theoverall time it takes to get a response (from an IP host 150) to thedata the IP host 110 sends.

[0072] A specific example in which this technique is useful relates toan Internet web page access application. With three-way handshakespoofing, an IP host's request to retrieve a web page can be on its wayto a web server without waiting for the end-to-end establishment of theTCP connection, thereby reducing the time it takes to download the webpage.

[0073] With Local Data Acknowledgement, the TSK 280 in the networkgateway 120 (for example) locally acknowledges data segments receivedfrom the IP host 110. This allows the sending IP host 110 to sendadditional data immediately. More importantly, TCP uses receivedacknowledgements as signals for increasing the current TCP window size.As a result, local sending of the acknowledgements allows the sending IPhost 110 to increase it TCP window at a much faster rate than supportedby end-to-end TCP acknowledgements. The TSK 280 (the spoofer) takes onthe responsibility for reliable delivery of the data which it hasacknowledged.

[0074] In the BPK 282, multiple TCP connections are multiplexed onto andcarried by a single backbone connection. This improves systemperformance by allowing the data for multiple TCP connections to beacknowledged by a single backbone connection acknowledgement (ACK),significantly reducing the amount of acknowledgement traffic required tomaintain high throughput across the backbone link 130. In addition, theBPK 282 selects a backbone connection protocol that is optimized toprovide high throughput for the particular link. Different backboneconnection protocols can be used by the BPK 282 with different backbonelinks without changing the fundamental TCP spoofing implementation. Thebackbone connection protocol selected by the BPK 282 providesappropriate support for reliable, high-speed delivery of data over thebackbone link 130, hiding the details of the impairments (for examplehigh latency) of the link from the TCP spoofing implementation.

[0075] The multiplexing by the BPK 282 allows for the use of a backbonelink protocol which is individually tailored for use with the particularlink and provides a technique to leverage the performance of thebackbone link protocol with much less dependency upon the individualperformance of the TCP connections being spoofed than conventionalmethods. Further, the ability to tailor the backbone protocol fordifferent backbone links makes the present invention applicable to manydifferent systems.

[0076] The PEP 200 may optionally include a data compression kernel 290for compressing TCP data and an encryption kernel 292 for encrypting TCPdata. Data compression increases the amount of data that can be carriedacross the backbone connection. Different compression algorithms can besupported by the data compression kernel 290 and more than one type ofcompression can be supported at the same time. The data compressionkernel 290 may optionally apply compression on a per TCP connectionbasis, before the TCP data of multiple TCP connections is multiplexedonto the backbone connection or on a per backbone connection basis,after the TCP data of multiple TCP connections has been multiplexed ontothe backbone connection. Which option is used is dynamically determinedbased on user configured rules and the specific compression algorithmsbeing utilized. Exemplary data compression algorithms are disclosed inU.S. Pat. Nos. 5,973,630, 5,955,976, the entire contents of which arehereby incorporated by reference. The encryption kernel 292 encrypts theTCP data for secure transmission across the backbone link 130.Encryption may be performed by any conventional technique. It is alsounderstood that the corresponding spoofer (in the example outlinedabove, the network gateway 140) includes appropriate kernels fordecompression and decryption, both of which may be performed by anyconventional technique.

[0077] The PK 284 provides prioritized access to the backbone linkcapacity. For example, the backbone connection can actually be dividedinto N (N>1) different sub-connections, each having a different prioritylevel. In one exemplary embodiment, four priority levels can besupported. The PK 284 uses user-defined rules to assign differentpriorities, and therefore different sub-connections of the backboneconnection, to different TCP connections. It should be noted that PK 284may also prioritize non-TCP traffic (e.g., UDP (User Datagram Protocol)traffic) before sending the traffic across the backbone link 130.

[0078] The PK 284 also uses user-defined rules to control how much ofthe backbone link 130 capacity is available to each priority level.Exemplary criteria which can be used to determine priority include thefollowing: Destination IP address; Source IP address; IP next protocol;TCP port numbers (which may apply to both the TCP destination and sourceport numbers); UDP port numbers (which may apply to both the UDPdestination and source port numbers); and IP differentiated services(DS) field. The type of data in the TCP data packets may also be used asa criterion. For example, video data could be given highest priority.Mission critical data could also be given high priority. As withselective TCP spoofing, any field in the IP packet can be used by PK 284to determine priority. However, it should be noted that under somescenarios the consequence of using such a field may cause different IPpackets of the same flow (e.g., TCP connection) to be assigned differentpriorities; these scenarios should be avoided.

[0079] As mentioned above, in addition to supporting selectiveprioritization rules for each of these criteria, AND and OR combinationoperators can be used to link criteria together. For example, using theAND combination operator, a rule can be defined to assign a priority forSNMP data received from a specific host. Also, the order in which therules are specified may be significant. It is possible for a connectionto match the criteria of multiple rules. Therefore, the PK 284 can applyrules in the order specified by the operator, taking the action of thefirst rule that matches. A default rule may also be set which definesthe action to be taken for IP packets which do not match any of thedefined rules. The set of rules selected by the operator may be definedin a prioritization profile.

[0080] As regards the path selection functionality, the PSK 286 isresponsible for determining which path an IP packet should take to reachits destination. The path selected by the PSK 286 can be determined byapplying path selection rules. The PSK 286 also determines which IPpackets should be forwarded using an alternate path and which IP packetsshould be dropped when one or more primary paths fail. Path selectionparameters can also be configured using profiles. The path selectionrules may be designed to provide flexibility with respect to assigningpaths while making sure that all of the packets related to the sametraffic flow (e.g., the same TCP connection) take the same path(although it is also possible to send segments of the same TCPconnection via different paths, this segment “splitting” may havenegative side effects). Exemplary criteria that can be used to select apath include the following: priority of the IP packet as set by the PK284 (should be the most common criterion), Destination IP address;Source IP address; IP next protocol; TCP port numbers (which may applyto both the TCP destination and source port numbers); UDP port numbers(which may apply to both the UDP destination and source port numbers);and IP differentiated services (DS) field. Similar to selective TCPspoofing and prioritization, the PSK 284 may determine a path by usingany field in the IP packet.

[0081] As with the prioritization criteria (rules) the AND and ORcombination operators can be used to link criteria together. Forexample, using the AND combination operator, a rule can be defined toselect a path for SNMP data received from a specific host. Also, theorder in which the rules are specified may be significant. It ispossible for a connection to match the criteria of multiple rules.Therefore, the PSK 286 can apply rules in the order specified by theoperator, taking the action of the first rule that matches. A defaultrule may also be set which defines the action to be taken for IP packetswhich do not match any of the defined rules. The set of rules selectedby the operator may be defined in a path selection profile.

[0082] By way of example, a path selection rule may select the pathbased any of the following path information in which IP packets matchthe rule: a primary path, a secondary path, and a tertiary path. Theprimary path is be specified in any path selection rule. The secondarypath is used only when the primary path has failed. If no secondary pathis specified, any IP packets that match the rule can be discarded whenthe primary path fails. The tertiary path is specified only if asecondary path is specified. The tertiary path is selected if both theprimary and secondary paths have failed. If no tertiary path isspecified, any IP packets that match the rule can be discarded when boththe primary and secondary paths fail. Path selection may be generalizedsuch that the path selection rule can select up to N paths where the Nthpath is used only if the (N−1)th path fails. The example above where N=3is merely illustrative, although N is typically a fairly small number.

[0083] By way of example, the operation of the system 100 is describedas follows. First, a backbone connection is established between the PEPs200 of two network gateways 120, 140 (i.e., the two spoofers), locatedat each end of the backbone link 130 for which TCP spoofing is desired.Whenever an IP host 110 initiates a TCP connection, the TSK 280 of thePEP 200 local to the IP host 110 checks its configured selective TCPspoofing rules. If the rules indicate that the connection should not bespoofed, the PEP 200 allows the TCP connection to flow end-to-endunspoofed. If the rules indicate that the connection should be spoofed,the spoofing PEP 200 locally responds to the IP host's TCP three-wayhandshake. In parallel, the spoofing PEP 200 sends a message across thebackbone link 130 to its partner network gateway 140 asking it toinitiate a TCP three-way handshake with the IP host 150 on its side ofthe backbone link 130. Data is then exchanged between the IP host 110,150 with the PEP 200 of the network gateway 120 locally acknowledgingthe received data and forwarding it across the backbone link 130 via thehigh-speed backbone connection, compressing the data as appropriatebased on the configured compression rules. The priority of the TCPconnection is determined when the connection is established. The BPK 282can multiplex the connection with other received connections over asingle backbone connection, the PK 284 determines the priority of theconnection and the PSK 286 determines the path the connection is totake.

[0084] The PEP 200, as described above, advantageously improves networkperformance by allocating TCP spoofing-related resources, such as bufferspace, control blocks, etc., only to TCP connections for which spoofingis beneficial; by spoofing the three-way handshake to decrease dataresponse time; by reducing the number of ACKs which are transmitted byperforming local acknowledgement and by acknowledging multiple TCPconnections with a single ACK; by performing data compression toincrease the amount of data that can be transmitted; by assigningpriorities to different connections; and by defining multiple paths forconnections to be made.

[0085]FIG. 3 shows an exemplary stack which illustrates the relationshipbetween the TCP stack and the PEP kernels 280, 282, 284, 286 of thepresent invention. The TSK 280 is primarily responsible for functionsrelated to TCP spoofing. The TSK 280, in an exemplary embodiment,includes two basic elements: a transport layer that encompasses a TCPstack 303 and an IP stack 305; and a TCP spoofing application 301. Thetransport layer is responsible for interacting with the TCP stacks(e.g., 303) of IP hosts 110 connected to a local LAN interface 220 of aPEP 210.

[0086] The TSK 280 implements the TCP protocol which includes theappropriate TCP state machines and terminates spoofed TCP connections.The TCP spoofing application 301 rests on top of the transport layer andact as the application that receives data from and sends data to the IPhosts 110 applications. Because of the layered architecture of theprotocol, the TCP spoofing application 301 isolates the details of TCPspoofing from the transport layer, thereby allowing the transport layerto operate in a standard fashion.

[0087] As shown in FIG. 3, the TCP spoofing application 301 can alsointerface to the BPK 282 associated with the WAN interfaces 230. The BPK282 performs backbone protocol maintenance, implementing the protocol bywhich the network gateways 120, 140 (in FIG. 1) communicate. The BPK 282provides reliable delivery of data, uses a relatively small amount ofacknowledgement traffic, and supports generic backbone use (i.e., usenot specific to the TSK 280); one such example is the reliable dataprotocol (RDP).

[0088] The BPK 282 lies above the PK 284 and the PSK 286, according toan exemplary embodiment. The PK 284 is responsible for determining thepriority of IP packets and then allocating transmission opportunitiesbased on priority. The PK 284 can also control access to buffer space bycontrolling the queue sizes associated with sending and receiving IPpackets. The PSK 286 determines which path an IP packet should take toreach its destination. The path selected by the PSK 286 can bedetermined applying path selection rules. PSK 286 may also determinewhich IP packet should be forwarded using an alternate path and whichpackets should be dropped when one or more primary paths fail.

[0089]FIGS. 4A and 4B show flow diagrams of the establishment of aspoofed TCP connection utilizing three-way handshake spoofing andwithout three-way handshake spoofing, respectively. The TCP SpoofingKernel 280 establishes a spoofed TCP connection when a TCP <SYN> segmentis received from its local LAN or a Connection Request message from itsTSK peer. It is noted that the three-way handshake spoofing may bedisabled to support an end to end maximum segment size (MSS) exchangewhich is more fully described below. For the purpose of explanation, thespoofed TCP connection establishment process is described with respectto a local host 400, a local PEP end point 402, a remote PEP end point404, and a remote host 406. As mentioned previously, the TSK 280 withineach of the PEP end points 402 and 404 provides the spoofingfunctionality.

[0090] In step 401, the local host 400 transmits a TCP <SYN> segment tothe local PEP end point 402 at a local LAN interface 220. When a TCPsegment is received from the local LAN interface 220, the platformenvironment 402 determines whether there is already a TCP connectioncontrol block (CCB) assigned to the TCP connection associated with theTCP segment. If there is no CCB, the environment 402 checks whether theTCP segment is a <SYN> segment that is being sent to a non-localdestination. If so, the <SYN> segment represents an attempt to bring upa new (non-local) TCP connection, and the environment 402 passes thesegment to the TCP Spoofing Kernel 280 to determine the TCP connection'sdisposition. When a TCP <SYN> segment is received from the local LANinterface 220 for a new TCP connection, the TCP Spoofing Kernel 280first determines if the connection should be spoofed. If the connectionshould be spoofed, TSK 280 uses (in an exemplary embodiment) thepriority indicated in the selected TCP spoofing parameter profile andthe peer index (provided by the environment 210 with the TCP <SYN>segment) to construct the handle of the backbone connection which shouldbe used to carry this spoofed TCP connection. In the exemplaryembodiment, the peer index is used as the 14 high order bits of thehandle and the priority is used as the two low order bits of the handle.The backbone connection handle is then used (via the TSK control block(TCB) mapping table) to find the TCB associated with the backboneconnection. TSK 280 of PEP end point 402 then checks whether thebackbone connection is up. If the backbone connection is up, TSK 280determines whether the number of spoofed TCP connections that arealready using the selected backbone connection is still currently belowthe CCB resource limit. The CCB resource limit is the smaller of thelocal number of CCBs (provided as a parameter by the platformenvironment 210) and the peer number of CCBs (received in the latest TSKpeer parameters (TPP) message from the TSK peer) available for thisbackbone connection. If the number of connections is still below thelimit, TSK 280 of PEP end point 402 assigns a unique TCP connectionidentifier (e.g., a free CCB mapping table entry index) to theconnection and calls the environment 210 to allocate a TCP connectioncontrol block for the connection.

[0091] TSK 280 of PEP end point 402 returns the TCP <SYN> segment backto the environment 210 to be forwarded unspoofed if any of the abovechecks fail. In other words, the following conditions result in the TCPconnection being unspoofed. First, if the selective TCP spoofing rulesindicate that the connection should not be spoofed. Also, there is nobackbone connection for the priority at which the TCP connection shouldbe spoofed (indicated by the absence of a TCB for the backboneconnection). No spoofing is performed if the backbone connection isdown. Additionally, if the number of spoofed TCP connections that arealready using the backbone connection reaches or exceeds a predeterminedthreshold, then no spoofing is performed. Further, if there is no CCBmapping table entry available or there is no CCB available from the CCBfree pool, then the TCP connection is forwarded unspoofed. For the casein which there is no backbone connection, TSK 280 of PEP end point 402may also post an event to alert the operator that there is a mismatchbetween the configured TCP spoofing parameter profiles and theconfigured set of backbone connections.

[0092] Continuing with the example, if all of the above checks pass, TSK280 of PEP end point 402 writes the backbone connection handle into thebuffer holding the TCP <SYN> segment. It is noted that this is not doneuntil a CCB is successfully allocated by the platform environment 402,because the environment does not count the buffer unless a CCB issuccessfully allocated. TSK 280 then copies the parameters from theselected TCP spoofing parameter profile into the CCB. Consequently,relevant information (e.g., the maximum segment size that is advertisedby the host (if smaller than the configured MSS), the initial sequencenumber, and etc.) is copied out of the TCP <SYN> segment and stored inthe CCB. It is noted that the source and destination IP addresses andsource and destination TCP port numbers will already have been placedinto the CCB by the platform environment 402 when the CCB was allocated;the environment 402 uses this information to manage CCB hash functioncollisions.

[0093] After allocating and setting up the CCB, the TCP Spoofing Kernel280 of PEP end point 402 constructs a Connection Request (CR) message,per step 403, and sends it to its TSK peer associated with the remotePEP end point 404. The CR message basically contains all of theinformation extracted from the TCP spoofing parameter profile and theTCP <SYN> segment and stored in the local CCB, e.g., the source anddestination IP addresses, the source and destination TCP port numbers,the MSS value, etc., with the exception of fields that have only localsignificance, such as the initial sequence number. (The IP addresses andTCP port numbers are placed into a TCP connection header.) In otherwords, the CR message contains all of the information that the peer TSKof PEP end point 404 requires to set up its own CCB. To complete thelocal connection establishment, the TCP Spoofing Kernel 280 of the localPEP end point 402 sends a TCP <SYN,ACK> segment to the local host 400 inresponse to the <SYN> segment received, per step 405. TSK 280 of PEP endpoint 402 performs step 405 simultaneously with the step of sending theConnection Request message (i.e., step 403), if three-way handshakespoofing is enabled. Otherwise, TSK 280 of 402 waits for a ConnectionEstablished (CE) message from its TSK peer of the remote PEP end point404 before sending the <SYN,ACK> segment. In an exemplary embodiment,TSK 280 of PEP end point 402 selects a random initial sequence number(as provided in IETF (Internet Engineering Task Force) RFC 793 which isincorporated herein by reference in its entirety) to use for sendingdata.

[0094] If three-way handshake spoofing is disabled, the MSS value sentin the <SYN,ACK> segment is set equal to the MSS value received in theCE message. If three-way handshake spoofing is enabled, the MSS value isdetermined from the TCP spoofing parameter profile selected for theconnection (and the configured path maximum transmission unit (MTU)).For this case, TSK 280 of PEP end point 402 then compares the MSS valuereceived in the Connection Established message, when it arrives, to thevalue it sent to the local host in the TCP <SYN,ACK> segment. If the MSSvalue received in the CE message is smaller than the MSS value sent tothe local host, a maximum segment size mismatch exists. (If an MSSmismatch exists, TSK may need to adjust the size of TCP data segmentsbefore sending them.) After sending the TCP <SYN,ACK> segment (step405), TSK 280 of the local PEP end point 402 is ready to start acceptingdata from the local host 400. In step 407, the local host 400 transmitsan <ACK> segment to the TSK 280 of PEP end point 402; thereafter, thelocal host forwards, as in step 409 data to the TSK 280 of PEP end point402 as well. When threeway handshake spoofing is being used, TSK 280does not need to wait for the Connection Established message to arrivefrom its TSK peer before accepting and forwarding data. As seen in FIG.4A, in step 411, TSK 280 of the local PEP end point 402 sends an <ACK>segment to the local host and simultaneously sends the TCP data (TD)from the local host 400 to the peer TSK of PEP end point 404 (per step413) prior to receiving a CE message from the peer TSK of PEP end point404.

[0095] However, TSK 280 of PEP end point 402 does not accept data fromits TSK peer of PEP end point 404 until after the CE message has beenreceived. TSK 280 of PEP end point 402 does not forward any datareceived from its TSK peer of PEP end point 404 to the local host 400until it has received the TCP <ACK> segment indicating that the localhost has received the <SYN,ACK> segment (as in step 407).

[0096] When a Connection Request message is received from a peer TSK(step 403), the TCP Spoofing Kernel 280 allocates a CCB for theconnection and then stores all of the relevant information from the CRmessage in the CCB. TSK 280 of PEP end point 404 then uses thisinformation to generate a TCP <SYN> segment, as in step 415, to send tothe remote host 406. The MSS in the <SYN> segment is set to the valuereceived from the TSK peer of PEP end point 404. When the remote hostresponds with a TCP <SYN,ACK> segment (step 417), TSK 280 of PEP endpoint 402 sends a Connection Established message to its TSK peer of theremote PEP end point 404 (step 419), including in the CE message the MSSthat is sent by the local host in the <SYN,ACK> segment. TSK 280 of PEPend point 402 also responds, as in step 421, with a TCP <ACK> segment tocomplete the local three-way handshake. The peer TSK of PEP end point404 then forwards the data that is received from TSK 280 to the host,per step 423. Concurrently, in step 425, the remote host 406 sends datato the peer TSK of PEP end point 404 which acknowledges receipt of thedata by issuing an <ACK> segment to the remote PEP end point 404, perstep 427. Simultaneously with the acknowledgement, the data is sent toTSK 280 of PEP end point 402 (step 429).

[0097] At this point, TSK 280 is ready to receive and forward data fromeither direction. TSK 280 forwards the data, as in step 431 to the localhost which, in turn, sends an <ACK> segment (step 433). If the dataarrives from its TSK peer before a <SYN,ACK> segment response isreceived from the local host, the data is queued and then sent after the<ACK> segment is sent in response to the <SYN,ACK> segment (when itarrives).

[0098] Turning now to FIG. 4B, a spoofed TCP connection is establishedwith the three-way handshake spoofing disabled. Under this scenario, thelocal host 400 transmits a TCP <SYN> segment, as in step 451, to the TSK280 within the local PEP end point 402. Unlike the TCP connectionestablishment of FIG. 4A, the local PEP end point 402 does not respondto the a TCP <SYN> segment with a <SYN,ACK> segment, but merely forwardsa CR message to the remote PEP end point 404 (step 453). Next, in step455, sends a TCP <SYN> segment to the remote host 406. In response, theremote host 406 transmit a TCP <SYN,ACK> segment back to the remote PEPend point 404 (per step 457). Thereafter, the remote PEP end point 404,as in step 459, forwards a CE message to the local PEP end point 402which subsequently issues a <SYN,ACK> segment to the local host 400, perstep 461. Simultaneous with step 459, the remote PEP end point 404issues an <ACK> segment to the remote host 406 (step 463).

[0099] Upon receiving the <ACK> segment, the remote host 406 may begintransmission of data, as in step 465. Once the PEP end point 404receives the data from the remote host 406, the remote PEP end point 404simultaneously transmits, as in step 467, the TD message to the localPEP end point 402 and transmits an <ACK> segment to the remote host 406to acknowledge receipt of the data (step 469).

[0100] Because the local host 400 has received a <SYN,ACK> segment fromthe local PEP end point 402, the local host 400 acknowledges themessage, per step 471. Thereafter, the local host 400 transmits data tothe local PEP end point 402. In this example, before the local PEP endpoint 402 receives the data from the local host 400, the local PEP endpoint 402 forwards the data that originated from the remote host 406 viathe TD message (step 467) to the local host 400, per step 475.

[0101] In response to the data received (in step 473), the local PEP endpoint 402 issues an <ACK> segment, as in step 477, and forwards the datain a TD message to the remote PEP end point 404, per step 479. The localhost 400 responds to the received data of step 475 with an <ACK> segmentto the local PEP end point 402 (step 481). The remote PEP end point 404sends the data from the local host 400, as in step 483, upon receipt ofthe TD message. After receiving the data, the remote host 406acknowledges receipt by sending an <ACK> segment back to the remote PEPend point 404, per step 485.

[0102]FIG. 5 shows the flow of packets with the PEP architecture,according to one embodiment of the present invention. As shown, acommunication system 500 includes a hub site (or local) PEP end point501 that has connectivity to a remote site PEP end point 503 via abackbone connection. By way of example, at the hub site (or local site)and at each remote site, PEP end points 501 and 503 handle IP packets.PEP end point 501 includes an Internal IP packet routing module 501 athat receives local IP packets and exchanges these packets with a TSK501 b and a BPK 501 c. Similarly, the remote PEP end point 503 includesan internal IP packet routing module 503 a that is in communication witha TSK 503 b and a BPK 503 c. Except for the fact that the hub site PEPend point 501 may support many more backbone protocol connections than aremote site PEP end point 503, hub and remote site PEP processing issymmetrical.

[0103] For local-to-WAN traffic (i.e., upstream direction), the PEP endpoint 501 receives IP packets from its local interface 220 (FIG. 2).Non-TCP IP packets are forwarded (as appropriate) to the WAN interface230 (FIG. 2). TCP IP packets are internally forwarded to TSK 501 b. TCPsegments which belong to connections that are not be spoofed are passedback by the spoofing kernel 501 b to the routing module 501 a to beforwarded unmodified to the WAN interface 230. For spoofed TCPconnections, the TCP spoofing kernel 501 a locally terminates the TCPconnection. TCP data that is received from a spoofed connection ispassed from the spoofing kernel 501 a to the backbone protocol kernel501 c, and then multiplexed onto the appropriate backbone protocolconnection. The backbone protocol kernel 501 c ensures that the data isdelivered across the WAN.

[0104] For WAN-to-local traffic (i.e., downstream direction), the remotePEP end point 503 receives IP packets from its WAN interface 230 (FIG.2). IP packets that are not addressed to the end point 503 are simplyforwarded (as appropriate) to the local interface 220 (FIG. 2). IPpackets addressed to the end point 503 which have a next protocol headertype of “PBP” are forwarded to the backbone protocol kernel 503 c. Thebackbone protocol kernel 503 c extracts the TCP data and forwards it tothe TCP spoofing kernel 503 b for transmission on the appropriatespoofed TCP connection. In addition to carrying TCP data, the backboneprotocol connection is used by the TCP spoofing kernel 501 b to sendcontrol information to its peer TCP spoofing kernel 503 b in the remotePEP end point 503 to coordinate connection establishment and connectiontermination.

[0105] Prioritization maybe applied at four points in the system 500within routing 5O1 a and TSK 501 b of PEP end point 501, and withinrouting 503 a, and TSK 503 b of PEP end point 503. In the upstreamdirection, priority rules are applied to the packets of individual TCPconnections at the entry point to the TCP spoofing kernel 501 b. Theserules allow a customer to control which spoofed applications have,higher and lower priority access to spoofing resources. Upstreamprioritization is also applied before forwarding packets to the WAN.This allows a customer to control the relative priority of spoofed TCPconnections with respect to unspoofed TCP connections and non-TCPtraffic (as well as to control the relative priority of these othertypes of traffic with respect to each other). On the downstream side,prioritization is used to control access to buffer space and otherresources in the PEP end point 503, generally and with respect to TCPspoofing.

[0106] At the hub (or local) site, the PEP end point 501 may beimplemented in a network gateway (e.g. an IP Gateway), according to oneembodiment of the present invention. At the remote site, the PEP endpoint 503 may be implemented in the remote site component, e.g. asatellite terminal such as a Multimedia Relay, a Multimedia VSAT or aPersonal Earth Station (PES) Remote.

[0107] The architecture of system 500 provides a number of advantages.First, TCP spoofing may be accomplished in both upstream and downstreamdirections. Additionally, the system supports spoofing of TCP connectionstartup, and selective TCP spoofing with only connections that canbenefit from spoofing actually spoofed. Further, system 500 enablesprioritization among spoofed TCP connections for access to TCP spoofingresources (e.g., available bandwidth and buffer space). Thisprioritization is utilized for all types of traffic that compete forsystem resources.

[0108] With respect to the backbone connection, the system 500 issuitable for application to a satellite network as the WAN. That is, thebackbone protocol is optimized for satellite use in that control blockresource requirements are minimized, and efficient error recovery fordropped packets are provided. The system 500 also provides a feedbackmechanism to support maximum buffer space resource efficiency. Further,system 500 provides reduced acknowledgement traffic by using a singlebackbone protocol ACK to acknowledge the data of multiple TCPconnections.

[0109]FIG. 6 illustrates the flow of IP packets through a PEP end point,according to an embodiment of the present invention. When IP packets arereceived at the local LAN interface 220, the PEP end point 210determines (as shown by decision point A), whether the packets aredestined for a host that is locally situated; if so, the IP packets areforwarded to the proper local LAN interface 220. If the IP packets aredestined for a remote host, then the PEP end point 210 decides, perdecision point B, whether the traffic is a TCP segment. If the PEP endpoint 210 determines that in fact the packets are TCP segments, then theTSK 280 determines whether the TCP connection should be spoofed.However, if the PEP end point 210 determines that the packets are notTCP segments, then the BPK 282 processes the traffic, along with the PK284 and the PSK 286 for eventual transmission out to the WAN. It shouldbe noted that the BPK 282 does not process unspoofed IP packets; i.e.,the packets flow directly to PK 284. As seen in FIG. 6, traffic that isreceived from the WAN interface 230 is examined to determine whether thetraffic is a proper PBP segment (decision point D) for the particularPEP end point 210; if the determination is in the affirmative, then thepackets are sent to the BPK 282 and then the TSK 280.

[0110] Routing support includes routing between the ports of the PEP EndPoint 210 (FIG. 2), e.g., from one Multimedia VSAT LAN port to another.Architecturally, the functionalities of TCP spoofing, prioritization andpath selection, fit between the IP routing functionality and the WAN.PEP functionality need not be applied to IP packets which are routedfrom local port to local port within the same PEP End Point 210. TCPspoofing, prioritization and path selection are applied to IP packetsreceived from a local PEP End Point interface that have been determinedto be destined for another site by the routing function.

[0111]FIG. 7 shows the relationship between PEP End Points and PEP EndPoint profiles, in accordance with an embodiment of the presentinvention. PEP parameters are primarily configured via a set of profiles701 and 703 which are associated with one or more PEP end points 705. Inan exemplary embodiment, PEP parameters are configured on a per PEP EndPoint basis, such as whether TCP spoofing is globally enabled. Theseparameters are configured in the PEP End Point profiles 701 and 703. Itis noted that parameters that apply to specific PEP kernels may beconfigured via other types of profiles. Profiles 701 and 703 are anetwork management construct; internally, a PEP End Point 705 processesa set of parameters that are received via one or more files.

[0112] Whenever the PEP End Point 705 receives new parameters, theplatform environment compares the new parameters to the existingparameters, figures out which of the PEP kernels are affected by theparameter changes, and then passes the new parameters to the affectedkernels. In an exemplary embodiment, all parameters are installeddynamically. With the exception of parameters that are componentspecific (such as the IP addresses of a component), all parameters maybe defined with default values.

[0113] As mentioned previously, the PEP end point 210 may be implementedin a number of different platforms, in accordance with the variousembodiments of the present invention. These platforms may include an IPgateway, a Multimedia Relay, a Multimedia VSAT (Very Small ApertureTerminal), and a Personal Earth Station (PES) Remote, as shown in FIGS.8-11, respectively. In general, as discussed in FIG. 2, the PEP endpoint 210 defines a local LAN interface 220 an interface through whichthe PEP End Point 210 connects to IP hosts located at the site. A WANinterface 230 is an interface through which the PEP End Point 210connects to other sites. It is noted that a WAN interface 230 canphysically be a LAN port. FIGS. 8-11, below, describe the specific LANand WAN interfaces of the various specific PEP End Point platforms. Theparticular LAN and WAN interfaces that are employed depend on whichremote site PEP End Points are being used, on the configuration of thehub and remote site PEP End Points and on any path selection rules whichmay be configured.

[0114]FIG. 8 shows the interfaces of the PEP end point implemented as anIP gateway, according to one embodiment of the present invention. By wayof example, an IP Gateway 801 has a single local LAN interface which isan enterprise interface 803. The IP Gateway 803 employs two WANinterfaces 805 for sending and receiving IP packets to and from remotesite PEP End Points: a backbone LAN interface and a wide area access(WAA) LAN interface.

[0115] The backbone LAN interface 805 is used to send IP packets toremote site PEP End Points via, for example, a Satellite Gateway (SGW)and a VSAT outroute. A VSAT outroute can be received directly byMultimedia Relays (FIG. 9) and Multimedia VSATs (FIG. 10) (and is theprimary path used with these End Points); however, IP packets can besent to a PES Remote (FIG. 11) via a VSAT outroute.

[0116]FIG. 9 shows a Multimedia Relay implementation of a PEP end point,in accordance with an embodiment of the present invention. A MultimediaRelay has two or three local LAN interfaces 903. Additionally, theMultimedia Relay 901 has up to two WAN interfaces 905 for sending IPpackets to hub site PEP End Points: one of its LAN interfaces and a PPPserial port interface, and four or five interfaces for receiving IPpackets from hub site PEP End Points, a VSAT outroute, all of its LANinterfaces, and a PPP serial port interface. It is noted that a PPP(Point-to-Point Protocol) serial port interface and a LAN interface aregenerally not be used at the same time.

[0117] A Multimedia Relay 901 supports the use of all of its LANinterfaces 903 at the same time for sending and receiving IP packets toand from hub site PEP End Points. Further, a Multimedia Relay 905supports the use of a VADB (VPN Automatic Dial Backup) serial portinterface for sending and receiving IP packets to and from the hub sitePEP End Points.

[0118]FIG. 10 shows a Multimedia VSAT implementation of the PEP endpoint, according to one embodiment of the present invention. AMultimedia VSAT 1001, in an exemplary embodiment, has two local LANinterfaces 1003. Support for one or more local PPP serial portinterfaces may be utilized. The Multimedia VSAT 1001 has two WANinterfaces 1005 for sending IP packets to hub site PEP End Points: aVSAT inroute and one of its LAN interfaces. The Multimedia VSAT 1001thus has three interfaces for receiving IP packets from hub site PEP EndPoints, the VSAT outroute and both of its LAN interfaces 1003. AMultimedia VSAT 1003 may support uses of both of its LAN interfaces 1003at the same time for sending and receiving IP packets to and from hubsite PEP End Points. The Multimedia VSAT 1003 further supports the useof a VADB serial port interface for sending and receiving IP packets toand from the hub site PEP End Points.

[0119]FIG. 11 shows a PES Remote implementation of a PEP end point,according to one embodiment of the present invention. A PES Remote 1101may have a local LAN interface and/or several local IP (e.g. PPP, SLIP,etc.) serial port interfaces, collectively denoted as LAN interfaces1103. The particular LAN interfaces 1103 depend on the specific PESRemote platform. PES Remote 1101, in an exemplary embodiment, has up tofive WAN interfaces 1105 for sending IP packets to hub site PEP EndPoints, an ISBN inroute, a LAN interface, a VADB serial port interface,a Frame Relay serial port interface and an IP serial port interface, andup to five existing interfaces for receiving IP packets from hub sitePEP End Points: an ISBN outroute, a LAN interface, a VADB serial portinterface, a Frame Relay serial port interface, and an IP serial portinterface. The physical Frame Relay serial port interface may besupporting multiple Permanent Virtual Circuits (PVCs); some of which areequivalent to local interfaces 1103 and some of which are WAN interfaces1105.

[0120]FIG. 12 shows the flow of TCP spoofing buffers through a PEP EndPoint, according to an embodiment of the present invention. In thisexample, there are six logical buffer pools involved with receiving,processing and forwarding TCP segments for spoofed TCP connections: aLAN to WAN (L2W) buffer pool 1201; a WAN to LAN (W2L) buffer pool 1203;a LAN Receive (LAN Rx) buffer pool 1205; a LAN Transmit (LAN Tx) bufferpool 1207; a WAN Receive (WAN Rx) buffer pool 1209; and a WAN Transmit(WAN Tx) buffer pool 1211.

[0121] The interfaces and the buffer pools shown in FIG. 12 are logicalentities. It is noted that the buffer flow shown in FIG. 12 issimplified in some cases for the purpose of explanation; for example, “abuffer” may constitute multiple physical buffers. Physically, there maybe more than one LAN or WAN interface and, in some cases for someplatforms, the same physical interface may be used as both a LANinterface 1213 and a WAN interface 1215. The buffer pools 1201, 1203,1205, 1207, 1209, and 1211 are logical in that the same physical set ofbuffers maybe used to implement more than one of the buffer pools eitherfor implementation convenience or because the LAN and WAN interfaces1213, 1215 are the same physical interface. Details on the platformspecific physical implementation of the logical buffer pools 1201, 1203,1205, 1207, 1209, and 1211 are described below.

[0122] When an IP packet arrives from the local LAN, the LAN interface1213 receives the packet into a buffer from the LAN Rx buffer pool 1205and passes the packet to the platform environment 210. The platformenvironment 210 copies the IP packet from the LAN Rx buffer 1205 into aLAN to WAN buffer 1201, and then returns the LAN Rx buffer 1205 to theLAN interface 1213. In a platform where the LAN Rx buffer 1205 and LANto WAN buffer 1201 are physically the same, the environment 210 mayavoid the copy and simply exchange a LAN to WAN buffer 1201 for the LANRx buffer 1205. Whether or not an actual copy occurs, if no LAN to WANbuffer 1201 is available, the IP packet is discarded (by returning theoriginal LAN Rx buffer 1205 to the LAN interface 1213) and must berecovered from in the same manner as if the IP packet was lost crossingthe LAN.

[0123] The environment 210 passes IP packets that contain spoofed TCPsegments to the TCP Spoofing Kernel 280 (when TCP spoofing is enabled).The LAN to WAN buffer 1201 handling of IP packets that do not containTCP segments is described below. The environment 210 recognizes a TCPspoofed TCP segment by the presence of a CCB for the segment. Theenvironment 210 also passes TCP <SYN> segments to TSK 280 to determinewhether a new connection should be spoofed. If the TCP <SYN> segmentdoes not belong to a TCP connection which should be spoofed, TSK 280returns the IP packet to the environment 210 with an indication toforward the TCP segment unspoofed. There are also circumstances in whichTSK 280 may return a TCP segment to be forwarded unspoofed even whenthere is a CCB for the TCP connection. If the TCP segment does belong toa TCP connection which is being (or is about to be) spoofed, TSK 280processes the TCP segment and then either forwards the contents of theTCP segment to its TSK 280 peer or discards it and returns the buffer ofthe segment to the platform environment 210. The platform environment210, in turn, returns the buffer to the LAN to WAN buffer pool 1201. Insome cases, TSK 280 does not need to forward the received TCP segmentbut does need to send a TSK message (as a consequence of receiving theTCP segment) to its TSK peer. (For example, when a TCP <SYN> segment isreceived, the <SYN> segment is not forwarded to the TSK peer but aConnection Request message may need to be sent to the TSK peer.) Whenthis is the case, rather than discard the TCP segment's buffer and thenask for a new buffer to generate the TSK message which needs to be sent,TSK 280 simply reuses the buffer in which the TCP segment was received.

[0124] For cases where TSK 280 needs to send a TSK message to its peerasynchronous to the reception of a TCP segment, TSK 280 requests a LANto WAN buffer 1201 from the platform environment 210 and uses thisbuffer 1201 to construct the message. To forward a data or control TSKmessage to its TSK peer, the TCP Spoofing Kernel 280 passes the bufferof the message (along with an indication of which backbone connectionshould be used to send the message) to the Backbone Protocol Kernel 282.Once a message has been passed to BPK 282, BPK 282 assumes ownership ofthe message's LAN to WAN buffer 1201. TSK messages are sent by BPK 282to its BPK peer as PBP segments. To send a PBP segment, BPK 282 passesthe segment as an IP packet to the platform environment 210 fortransmission on the appropriate WAN interface 1215. The environment 210passes the IP packet to the appropriate WAN interface 1215, copying theLAN to WAN buffer 1201 into a WAN Tx buffer 1211.

[0125] Because BPK 282 needs to provide guaranteed delivery of TSKmessages, BPK 282 must get back and hold for potential retransmissionany TSK messages it transmits. Therefore, (when requested via a flagused with the interface,) the platform environment 210 must return an IPpacket passed to it back to BPK 282 after it has been transmitted. It isnoted that when the environment 210 returns IP packets to BPK 282, for agiven backbone connection, the environment must return the IP packets toBPK 282 in the order that they were given to it by BPK 282. According toan exemplary embodiment, this may be accomplished automatically byperforming an immediate copy into a WAN Tx buffer 1211. Alternatively,this may be performed through the use of a queuing mechanism to ensurethat the packets are returned in order. In a platform 210 which uses aLAN to WAN buffer 1201 and a WAN Tx buffer 1211 are compatible, theenvironment 210 may opt to not do an actual copy when BPK 282 does notwant the IP packet back. If the buffers 1201 and 1211 are compatible,the allocated WAN Tx buffer 1211 can be returned to the LAN to WANbuffer pool 1201 with the LAN to WAN buffer 1201 forwarded as a WAN Txbuffer 1211.

[0126] The Backbone Protocol Kernel 282 can also generate segments to besent to its BPK peer without receiving a message from TSK 280, e.g., tosend an acknowledgement for PBP segments which have been received. Tosend such a segment, BPK 282 allocates a buffer from the LAN to WANbuffer pool 1201 (via the platform environment 210), constructs the PBPsegment it needs to send and then forwards the segment as an IP packetto the platform environment 210 in the same way that it forwards PBPsegments which contain TSK messages. It is noted that the allocation ofbuffers to send PBP acknowledgements occurs independently from thereception of PBP segments. BPK 282 will still process any received PBPsegment even if no LAN to WAN buffer 1201 is available to send aresponse to the segment. The lack of a buffer to send a response issimply recovered from in the same manner as if the segment wassuccessfully transmitted but lost crossing the WAN. After the BackboneProtocol Kernel is done with a segment it has transmitted, e.g. it hasreceived an acknowledgement for the segment from its BPK peer, itreturns the buffer of the segment to the LAN to WAN buffer pool 1201.

[0127] Losing a received or transmitted TCP segment or PBP segmentbecause a buffer is unavailable is not critical. The lost IP packet canbe recovered from in the same manner as if the IP packet had been lostcrossing the LAN or WAN. However, not being able to send a TSK messagebecause a buffer is unavailable presents a more serious situation. TSK280 assumes that messages cannot be lost in the pipe provided betweenitself and its peer by the PEP Backbone Protocol. Therefore, specialhandling is required if TSK 280 attempts to generate a TSK message fromscratch and is unable to do so. In some cases, for example, thegeneration a TSK Peer Parameters message, the appropriate reaction is tostart a timer and reattempt to send the message when the timer goes off.In other cases, for example, the inability to send a ConnectionTerminated message, the appropriate reaction might be to disregard theevent which required the CT message to be sent. For example, if themessage is being sent due to a timeout, the timer can be restarted withsome short value and reprocessed when it expires again.

[0128] When an IP packet arrives from the WAN, the WAN interface 1215receives the packet into a buffer from the WAN Rx buffer pool 1209 andpasses it to the platform environment 210. The platform environment 210copies the IP packet from the WAN Rx buffer 1209 into a WAN to LANbuffer 1203 and then returns the WAN Rx buffer 1209 to the WAN interface1215. In a platform 210 in which the WAN Rx buffer 1209 and WAN to LANbuffer 1203 are physically the same, the environment 210 may avoid thecopy and simply exchange a WAN to LAN buffer 1203 for the WAN Rx buffer1209. Whether or not an actual copy occurs, if no WAN to LAN buffer 1203is available, the IP packet is discarded (by returning the original WANRx buffer 1209 to the WAN interface) and must be recovered from in thesame manner as if the IP packet was lost crossing the WAN. Theenvironment 210 passes all IP packets which contain PBP segments(addressed to this PEP end point 210) to the Backbone Protocol Kernel282. The WAN to LAN buffer handling of other types of IP packets isdescribed below.

[0129] BPK handling of PBP segments depends on the type of PBP segment.In terms of buffer handling, there are two types of PBP segments: (1)PBP segments which can be immediately processed and discarded, i.e. PBPcontrol segments; and (2) PBP segments which must be forwarded to theTCP Spoofing Kernel 280, i.e. TSK messages. For a PBP control segment,e.g., a PBP segment used to bring up backbone connections, the BackboneProtocol Kernel 282 can take whatever actions are required by thesegment and then return the buffer of the segment to the WAN to LANbuffer pool 1203. BPK 282 forwards received TSK messages to the TCPSpoofing Kernel 280. Once BPK 282 has passed a message to TSK 280, TSK280 assumes ownership of the message's WAN to LAN buffer 1203. TSK WANto LAN buffer handling is described below. It is noted that a segmentcontaining a TSK message does not necessarily need to be forwarded toTSK 280 immediately. Out of sequence segments are held by BPK 282 on abackbone connection's resequencing queue while BPK 282 waits for themissing segments. (BPK 282 must forward TSK messages in order to the TCPSpoofing Kernel.) Also, the Backbone Protocol Kernel does not generatemessages to communicate information (e.g. backbone connection resets) tothe TCP Spoofing Kernel. Any information that BPK 282 needs to pass toTSK 280 is passed using a procedural interface. Therefore, BPK 282 neverneeds to allocate a WAN to LAN buffer 1203 for its own use.

[0130] The TCP Spoofing Kernel 280 receives two types of messages fromits TSK peer: control messages (e.g., Connection Request messages), anddata messages (e.g., TCP Data messages). Both types of messages can, insome cases, be immediately discarded by TSK 280 (for example, uponreception of a TCP Data message for a connection which no longerexists.) This is accomplished simply by returning the buffer of themessage to the WAN to LAN buffer pool 1203. Generally, however,processing is required for a message received from a TSK peer. Controlmessages may require the generation of a corresponding TCP segment to besent to a local host. For example, the reception of a Connection Requestmessage will usually result in a TCP <SYN> segment being sent to a localhost. However, reception of a Connection Established message does notresult in a TCP <SYN,ACK> segment being sent to a local host if TSK 280has already sent the <SYN,ACK> segment. When a control message requiresthat a TCP segment be sent to a local host, TSK 280 stores anyinformation it requires from the control message and then uses the WANto LAN buffer 1203 of the control message to construct the TCP segmentwhich needs to be sent. Besides being more efficient, reusing the WAN toLAN buffer 1203 avoids error scenarios where no additional WAN to LANbuffer 1203 is available for the TCP segment which needs to begenerated. For a data message, the TCP Spoofing Kernel must firstconvert the TSK message into a TCP data segment. This is basically doneby replacing the PBP and TSK buffer headers with an appropriate TCPheader 1515, using the mechanism as described later.

[0131] After the TCP Spoofing Kernel 280 converts a TSK message into aTCP segment, TSK 280 sends the TCP segment to a local host by passingthe segment as an IP packet to the platform environment 210 fortransmission on the appropriate LAN interface 1213. The environment 210passes the IP packet to the LAN interface 1213 for transmission; this isexecuted by allocating a LAN Tx buffer 1207 and then copying the IPpacket from the WAN to LAN buffer 1203 to the LAN Tx buffer 1207. A copyis done because TSK 280 needs to provide guaranteed delivery of TCP datasegments and, therefore, must get back and hold for potentialretransmission many of the TCP data segments the TSK 280 transmits.Therefore, (when requested via a flag used with the interface,) theenvironment 210 returns the IP packets passed to it back to TSK 280after these packets have been transmitted. Copying the IP packet into aLAN Tx buffer 1207 allows the environment 210 to perform thisimmediately. If the environment 210 cannot allocate a LAN Tx buffer 1207to copy the IP packet into, the environment 210 must return the IPpacket to TSK 280 as if the IP packet had been transmitted. TSK 280 thenrecovers from the error in the same manner as if the IP packet had beenlost crossing the local LAN. It is noted that when the environment 210returns IP packets to TSK 280, for a given TCP connection, theenvironment 210 must return the IP packets to TSK 280 in the order thatthese packets were given to it by TSK 280. The immediate copy makesmeeting this requirement simple.

[0132] The TCP Spoofing Kernel 280 can also generate TCP segments to besent to a local host without receiving a message from its TSK peer,e.g., to send an acknowledgement for TCP data segments that have beenreceived. To send such a segment, TSK 280 allocates a buffer from theWAN to LAN buffer pool 1203, constructs the TCP segment that the TSK 280needs to send and then forwards the segment as an IP packet to theplatform environment 210 in the same way that it forwards TCP segmentsgenerated by a TSK message received from its TSK peer. It is noted thatthe allocation of buffers to send TCP data acknowledgements occursindependently from the reception of TCP segments. TSK 280 will stillprocess any received TCP segment, including data segments, even if noWAN to LAN buffer 1203 is available to send a response to the segment.The lack of a buffer to send a response is simply recovered from in thesame manner as if the segment was successfully transmitted but lostcrossing the local LAN. After the TCP Spoofing Kernel is done with asegment it has transmitted, e.g. it has received an acknowledgement forthe segment from the local host, it returns the buffer of the segment tothe WAN to LAN buffer pool 1203.

[0133]FIG. 13 shows a diagram of the buffer management for unspoofed TCPconnections and for non-TCP (e.g. UDP) traffic, according to anembodiment of the present invention. Buffer management in the unspoofedcase is similar to, but much simpler than, buffer management for spoofedTCP connections. As seen in FIG. 13, in the LAN to WAN direction, theplatform environment 210 copies received IP packets out of LAN Rxbuffers 1205 into LAN to WAN buffers 1201. Non-TCP IP packets areforwarded directly to the WAN interface 1215 without being passedthrough the TCP Spoofing Kernel or the Backbone Protocol Kernel.Unspoofed TCP IP packets are forwarded like non-TCP IP packets after TSK280 “rejects” them. (If TCP spoofing is globally disabled, theenvironment 210 does not bother to send the TCP IP packets through TSK280.). In the WAN to LAN direction, the process is similar. The platformenvironment 210 copies received IP packets out of WAN Rx buffers 1209into WAN to LAN buffers 1203 and then, for all IP packets which are notPBP IP packets containing one of the platform's IP addresses as thedestination address, forwards the packets to the (appropriate) LANinterface 1213, copying the IP packets into LAN Tx buffers 1207. In someplatforms, it may be possible for the platform environment 210 to copythe IP packets directly from WAN Rx to LAN Tx buffers 1207. There is noneed for these packets to be processed by any PEP kernel.

[0134] The backbone connection associated with a buffer is stored in thebuffer. When no backbone connection is associated with the buffer, avalue of 0xFFFF is used. For debugging purposes (and to keep the bufferhandling code symmetrical), the platform environment 210 may keep trackof the number of currently allocated buffers associated with “backboneconnection” 0xFFFF.

[0135]FIG. 14 is a diagram of a basic format of the buffers used toimplement the PEP functionality, in accordance with an embodiment of thepresent invention. A buffer 1400 includes a buffer header 1401whichcontains platform specific buffer fields, if any such fields exist. Theformat of (and even the existence of) these fields is only known to theplatform environment 210. Following the platform specific buffer header1401 is a PEP common buffer header 1403 which in an exemplary embodimentis about 30 to 44 bytes in length. The fields in this header 1403 areknown to and used by the PEP kernels. Buffer 1400 also includes aportion that is designated for the IP packet 1405 which in addition tothe PEP common buffer header 1403 constitutes the “payload” of thebuffer 1400.

[0136] Buffers 1400 are passed from the environment 210 to a kernel,from a kernel to another kernel, and from a kernel to the environment210 via a pointer to the beginning of the PEP common buffer header 1401.Any pointer adjustments that are required to account for a platformspecific buffer header 1401 are made by the platform environment 210.

[0137] The platform environment 210, according to an exemplaryembodiment, provides the task context in which the PEP kernels operate.Therefore, from the point of view of platform specific buffermanagement, the PEP platform environment 210 is the explicit owner ofall of the buffers that are allocated for use for PEP functionality.Buffer handling formalities with respect to (explicit) buffer ownership(if any exist for a particular platform) occur at the point when theplatform environment 210 receives or returns a buffer from or to outsideof the PEP context. Within the context of the platform environment 210task, a buffer is considered to be owned by whichever kernel currentlypossesses it. However, no formal buffer ownership transfer has to occur.The transfer of ownership can be implicit. For example, when the TCPSpoofing Kernel 280 passes a TSK message to the Backbone Protocol Kernel282 for transmission across a backbone connection, TSK 280 passesimplicit ownership of the buffer to BPK 282. In an exemplary embodiment,only the implicit owner of a buffer is allowed to access the buffer.Except for the case in which the specific context of an interface isdefined to allow it, a kernel should not assume that fields in a bufferhave not been changed if the kernel passes a buffer outside of its owncontext and then gets it back.

[0138]FIG. 15 shows a diagram of an IP packet that is used in the systemof FIG. 1. An IP packet 1500 has an IP header 1501 (as defined in IETFRFC 791 which is incorporated herein by reference in its entirety)followed by a payload 1503. IP header 1501 is generally 20 bytes inlength. The IP header 1501 can be greater than 20 bytes in length if IPheader options are used.

[0139] The size of the IP packet payload 1503 is determined by themaximum transmission unit (MTU) size of the network that is being usedto carried the IP packet. For instance, the MTU of an Ethernet link is1500 bytes, supporting an IP packet payload 1503 of up to 1480 bytes. Asshown in FIG. 15, the IP packet payload generally carries the “messageunit” of some higher layer protocol. These higher layer protocols mayinclude User Datagram Protocol (UDP) 1505, TCP 1507, and the PEPfeature, PBP 1509. UDP 1505 includes a UPD header 1511 and a payload1513 (which contains the data). Similarly, TCP 1507 provides a TCPheader 1515 and a data portion 1517. The PBP 1509 format, for example,houses a TSK message 1518 with a TSK header 1519 and a data 1521. TheTSK message 1518, in turn, constitutes the payload, or data, 1523 of aPBP segment. PBP 1509 also includes a PBP header 1525.

[0140] Buffers are passed between the environment 210 and the PEPkernels as IP packets. At the TSK/BPK interface, buffers are passedbetween TSK 280 and BPK 282 as TSK messages. The PEP common bufferheader 1403, as more fully described below, is used to pass theappropriate buffer payload at each interface.

[0141]FIG. 16 shows a diagram of a format of the PEP common bufferheader, according to an embodiment of the present invention. The commonbuffer header 1403 has three purposes: (1) to provide a mechanism forpassing buffers between the environment 210 and the PEP kernels andbetween the various PEP kernels themselves; (2) to provide a mechanismwhich supports the ability for IP, TCP, PBP and TSK headers to grow (andshrink) without requiring a shift of the data in an IP packet, therebysignificantly improving performance by avoiding data copies; and (3)provide space for owner specific per buffer fields (eliminating the needto allocate separate per buffer data structures). It is noted that theboundary between the owner specific “header” and the header growth“header” is somewhat arbitrary in that a kernel, if it needs to, can putowner specific fields into the header growth space (and vice versa), ifthey will fit. However, this can only be done within a kernel. Theboundary between the two “headers” must be respected by a kernel whenpassing a buffer to the environment or another kernel.

[0142] The PEP common buffer header 1403 includes a Flags+Offset field1601 which (by way of example) is 2 bytes in length, whereby 4 bits aredesignated for a Flags field 1601 a and the remaining 12 bits areprovided for the Payload Offset field 1601 b. With respect to the Flagsfield 1601 a, the first (most significant bit) flag bit holds thedirection (DIR)flag. The direction flag indicates whether thisparticular buffer has been allocated in the LAN to WAN direction (DIR=0)or WAN to LAN direction (DIR=1). The last (least significant bit) flagbit is reserved for use by the platform environment 210. The two middleflag bits are reserved. As regards the Payload Offset field 1601 b, thefield 1601 b specifies in bytes the current start of the buffer payload(e.g., IP packet). The header growth space in the buffer allows thisvalue to be adjusted both up and down. However, care must be taken tonot adjust the payload offset beyond the boundary between the ownerspecific field 1605 and header growth field 1607.

[0143] The Connection Handle field 1603 which is 2 Bytes in length,specifies the handle of the backbone connection to which this buffer hasbeen allocated. The connection handle may be set, for example, to 0xFFFFin buffers that do not contain spoofed TCP segments or PBP segments andin buffers for which the platform environment 210 has not yet determinedthe proper backbone connection to which to allocate the buffer. Thelatter applies to TCP <SYN> segments received from the local LAN.

[0144] The 24 byte Owner Specific “Header” field 1605 provides forshifting the contents of a buffer to accommodate different header sizesthat the CPU requires. If the payload of the buffer is small, the CPUrequired may not be significant. But, when the payload is large, e.g.,when the buffer contains user data, the CPU required can be verysignificant. And, since carrying user data is the real purpose of anetwork (and, ideally, represents the vast majority of the traffic,)optimizing for the case of large user data messages is desirable. Thesize of a received IP header 1501 and a transmitted IP header 1501 willgenerally be the same, i.e. 20 bytes. Therefore, in general, replacingone IP header 1501 with another requires no special buffer handling. Onthe other hand, the size of a TCP header 1515 differs from the size of aPBP header 1525 and even from the size of combined PBP and TSK headers.A TCP header 1515 is generally 20 bytes. The use of TCP options canincrease the size of the TCP header 1515. As currently defined, a PBPheader 1525 is 12 bytes when the PBP segment includes a TSK message. Inmost cases, a TSK header 1519 (FIG. 15) for a data message is 6 bytes.For the exceptions, the TSK header 1519 is 18 bytes. Therefore, thecombined PBP and TSK headers 1525 and 1519 for a data message are 18bytes most of the time.

[0145] On the surface, it might appear that changing either the PBPheader 1525 or TSK header 1519 so that the combined headers equal 20bytes to match the size of the TCP header 1515 may improve bufferhandling performance (at the expense of wasting a couple of bytes ofoverhead when sending PBP segments across the WAN). However, in additionto reducing flexibility regarding handling TCP options, when looked atmore closely, it is observed that this is not the case. The reason forthis is that TSK and BPK buffer handling occur independently. TSK 280 isnot aware of the size of the PBP header 1525 and should not be. And,conversely, BPK 282 is not aware of the size of the TSK header 1519 andshould not be. Making the kernels aware of each other's header sizesviolates their protocol layering relationship and would introduce anundesirable dependency between the kernels. The method defined to handlethis problem is to use extra space at the front of the buffer along witha “pointer” (i.e., an offset count) to the buffer payload (e.g., thecurrent start of the IP packet). This method allows the data to remainin place with only the buffer headers moved around. And, it takesadvantage of the fact that the PEP kernels generally only reuse thespace for headers. Fields in a header rarely remain unchanged and,therefore, a shift in the location of a header simply requires a changein the location where a kernel needs to fill in fields not an actualshift of header contents. For example, the IP header 1501 required bythe TCP Spoofing Kernel 280 to send and receive TCP data to and from thelocal host contains no field values in common with the IP header 1501required by the Backbone Protocol Kernel 282 to send and receive thesame data across the WAN. And, the TCP header 1515 used to send andreceive data to and from the local host is completely replaced by thePBP and TSK headers 1519 and 1509 used to send and receive the same dataacross the WAN (and vice versa). In an exemplary embodiment, in a bufferthat has not had any header adjustments, the payload offset may point 44bytes into the buffer at the start of an IP packet (because the buffer,in this example, is initialized with 16 bytes of header growth space).If a header needs to be inserted which is smaller than the header it isreplacing, then the kernel which is making the adjustment moves theheaders to the right, updating the payload field in the buffer. If aheader needs to be inserted which is larger than the header it isreplacing, then the kernel which is making the adjustment moves theheaders to the left, again updating the payload offset field 1601 b inthe buffer. Of course, as indicated above, even when no headeradjustments are required, payload offset adjustments may be requiredbecause IP packets are not the buffer “unit” passed at all interfaces.In particular, TSK messages are the buffer “unit” passed between the TCPSpoofing Kernel 280 and the Backbone Protocol Kernel 282.

[0146]FIGS. 17 through 20 show the use of header growth space for TCPdata segments, according to an embodiment of the present invention. InFIG. 17, a buffer containing a TCP data segment received from the localLAN is passed by the platform environment 210 to the TCP Spoofing Kernel280. TSK 280 removes the 20 byte IP header 1501 and 20 byte TCP header1515 and adds a 6 byte TSK header 1519, updating the payload offset 1601b from 44 to 78 (representing the size difference between the originaland new headers), and then passes the buffer to the Backbone ProtocolKernel 282 as a TSK message. BPK 282 adds a 20 byte IP header 1501 and a12 byte PBP header 1525 to the TSK message, updating the payload offset1601 b from 78 to 46, and then passes the buffer to the platformenvironment 210 for forwarding towards the WAN.

[0147]FIG. 18 illustrates the same buffer flow for the case where TSK280 needs to insert a 12 byte TCP connection header 1515 for the TCPdata segment in addition to the TSK header 1519.

[0148] In FIG. 19, a buffer containing a TSK Data message received fromthe WAN is passed by the platform environment 210 to BPK 282. BPK 282removes the 20 byte IP header 1501 and 12 byte PBP header 1525, updatingthe payload offset 1601 b from 44 to 76, and then passes the buffer toTSK 280. TSK 280 removes the 6 byte TSK header 1519 and adds a 20 byteIP header 1501 and a 20 byte TCP header 1515 to convert the TSK Datamessage into a TCP data segment, updating the payload offset 1601 b from76 to 42, and then passes the buffer to the platform environment 210 forforwarding towards the WAN.

[0149]FIG. 20 illustrates the same buffer flow for the case where TSK280 needs to also remove a 12 byte TCP connection header 1515 from theTSK Data message in addition to the TSK header 1519.

[0150] An initial size of 16 bytes may be selected because a 16 byteheader growth “header” provides 4 byte alignment and provides margin forunanticipated header growth requirements. However, in a particularplatform, the platform environment 210 may choose to use an initialheader growth space size of larger than 16 bytes. This might bedesirable, for example, to provide room for an Ethernet MAC header,potentially allowing the use of a common physical buffer pool to beshared by all of the logical buffer pools.

[0151] It is noted that not all of the TCP segments that are sent by theTCP Spoofing Kernel 280 originate from TSK messages received from a TSKpeer. TSK 280 often needs to generate a TCP segment (e.g., anacknowledgement for a received TCP data segment) to send to a local host“from scratch”. As is indicated previously, when TSK 280 needs togenerate such a TCP segment, TSK 280 calls the platform environment 210to allocate a WAN to LAN buffer 1203. The buffer 1203 that is providedby the environment 210 is initialized with its payload offset 1601 bpointing to the first byte beyond the platform's default header growth“header” (e.g., 44 bytes into the buffer). Because no headers need to beinserted in front of the headers that are inserted by TSK 280 (exceptfor the LAN MAC header inserted for all IP packets), TSK 280 need not beconcerned with leaving room for additional headers in the buffer. TSK280 can insert an IP header 1501 and a TCP header 1515 at the locationprovided by the platform environment 210. This is illustrated in FIG.21.

[0152] Similarly, not all of the PBP segments that are sent by theBackbone Protocol Kernel 282 originate from TSK messages that areforwarded by TSK 280. BPK 282 often needs to generate a PBP segment(e.g., an acknowledgement for a received PBP data segment) to send to aBPK peer “from scratch”. When BPK 282 needs to generate such a PBPsegment, BPK 282 calls the platform environment 210 to allocate a LAN toWAN buffer 1201. The buffer provided by the environment is initializedwith its payload offset pointing to the first byte beyond the platform'sdefault header growth “header” (e.g., 44 bytes into the buffer). Sinceno headers will need to be inserted in front of the headers inserted byBPK 282 (except for any WAN MAC header inserted for all IP packets), BPK282 does not need to worry about leaving room for additional headers inthe buffer and can insert an IP header 1501 and a PBP header 1525 at thelocation provided by the environment 210. This is illustrated in FIG.22.

[0153] BPK 282 never needs to generate messages to a local host (via TSK280). However, TSK 280 does need to generate TSK messages (e.g., aConnection Terminated message when a connection is terminated due toretransmission failures) to send to TSK peers (via BPK 282). When TSK280 needs to generate a TSK message, TSK 280 calls the platformenvironment 210 to allocate a LAN to WAN buffer 1201. As in the othercases described above, the buffer provided by the environment isinitialized with its payload offset 1601 b pointing to the first bytebeyond the platform's default header growth “header” (e.g., 44 bytesinto the buffer). However, because a TSK message will be forwarded to aTSK peer via BPK 282, TSK 280 must, in this case, leave room for BPK 282to insert PBP and IP headers; however, this does not require TSK 280 toknow anything about the size of the PBP header 1525. TSK 280 can simplyadd the TSK header 1519 (and TCP connection header 1515, if necessary)in the locations it would have done so if the buffer had been receivedwith an IP header 1501 and a TCP header 1515 in it, as shown in FIG. 23.

[0154] It is noted in FIG. 23 that the scenario whereby TSK 280 adjuststhe buffer to include an IP header 1501 and a TCP header 1515 so thatthe buffer looks the same as if the buffer had been received from thelocal LAN is merely illustrative. The implementation, for example, canimmediately place the TSK header 1519 in the right place by adding theright value to the payload offset 1601 b. When TSK 280 and BPK 282 callthe environment to allocate a buffer, they provide the size of thebuffer they wish to allocate. However, the size indicated only reflectsthe size of the segment or message they wish to generate (including therelevant protocol headers). The size does not include the PEP bufferheader or any platform specific buffer header required. The environment210 adjusts the requested size accordingly. This adjustment is left tothe environment 210 for two reasons. First, TSK 280 and BPK 282 have noknowledge of platform specific buffer headers. Second, a particularplatform environment 210 might want to use a larger header growth spacethan the required minimum.

[0155] Another buffer usage scenario exists that involves the TCPSpoofing Kernel is the case in which a “message” is received and notforwarded, but reception of the “message” requires that a different“message” be generated in the same direction (LAN to WAN or WAN to LAN)as that of the original “message”. For example, a received TCP <RST>segment is not forwarded, but it can result in the need to send aConnection Terminated TSK message and vice versa. When this occurs, TSK280 reuses the buffer of the original “message” to generate the new“message” rather than deallocate the received buffer and allocate a newone. However, this does not require any special handling because of thefact that TSK 280 already completely replaces the headers of TCPsegments and TSK messages it receives before forwarding. The same bufferpayload offset adjustments made for forwarded data “messages” will workwhen reusing a buffer. This is illustrated in FIGS. 24 and 25 which showthat the same header adjustment made in FIGS. 18 and 20 may be used; theonly difference is that there is no data in the buffer which needs to bemaintained for the reuse case.

[0156] As indicated previously, when TSK 280 or BPK 282 allocate abuffer to construct an IP packet to be sent, they specify the requiredsize of the buffer. BPK 282 does not generate IP packets that are to beforwarded towards the local LAN (by way of TSK 280) and, therefore, isnot concerned with leaving room for TSK 280 to insert data into theallocated buffer. However, TSK 280 does generate TSK messages to beforwarded towards the WAN (by way of BPK 282). Therefore, when TSK 280allocates a buffer for the purpose of sending a TSK message, TSK 280must leave room for the PBP header 1525. However, BPK 282 inserts thePBP header 1525 in front of the TSK header 1519, treating the TSKmessage as data.

[0157] Therefore, as long as TSK 280 follows the above strategy of“inserting” space for IP header 1501 and TCP header 1515, the size ofthe allocated buffer remains correct. However, the size of a buffer maynot be correct if the buffer is reused. For example, a received TCP<SYN> segment will usually be a 40 byte IP packet. But, the IP packetused for the TSK CR message which needs to be sent as a result ofreceiving the TCP <SYN> segment will be larger than 40 bytes. If avariable, exact size buffer strategy is in use in the PEP End Pointplatform 210, there will not be room in the buffer to build the CRmessage. There are two options to address this problem. The first optionis to not allow the reuse of a buffer for this case. TSK 280 could berequired to deallocate the original buffer and allocate a new, largerbuffer. The second option is to have the platform environment 210 alwaysallocate a buffer of at least some minimum size when a buffer isrequested or when the environment 210 copies a received TCP or PBPsegment out of a LAN Rx buffer 1205 or WAN Rx buffer 1209 into a LAN toWAN buffer 1201 or WAN to LAN buffer 1203. This is the approachadvantageously simplifies the PEP kernel code.

[0158] Even when a platform is using a fixed size buffer strategy, thereis still a need to enforce a minimum buffer size. In this case, theminimum buffer size is required to ensure that all of the fields whichneed to be accessed by a PEP kernel are in the first buffer holding anIP packet. This includes all of the protocol headers and the data forTSK control messages. This is true if the buffer strategy is to usesingle, fixed size buffers since this requires the use of large buffers.However, if buffer chaining is used, then the first buffer in the chainmust be large enough to hold all of the information which needs to beaccessed by the PEP kernels. For example, the minimum buffer size may bespecified as 100 bytes; i.e., a platform environment 210 must notallocate a buffer smaller than 100 bytes. The minimum value isconfigurable. Also, a platform environment 210 may use a minimum buffersize of greater than 100 bytes if desired; for example, to improvebuffer alignment efficiency. Enforcing the minimum buffer size is theresponsibility of the platform environment 210. The fact that the bufferreturned by the environment might be bigger than requested istransparent to TSK 280 and BPK 282.

[0159] The various PEP kernels often need to chain strings of bufferstogether to implement queues. This may be implemented by allocating asmall, separate buffer descriptor block for each buffer and then usingfields in the buffer descriptor to point to a buffer and to link bufferdescriptors together. However, since there is basically a one for onerelationship between the number of buffer descriptors required and thenumber of buffers required, an alternative approach is to basicallyembed the buffer descriptor in the buffer itself. This is the purpose ofthe owner specific part 1605 of the PEP common buffer header 1403. Theowner specific “header” 1605 is available to the current owner of abuffer to be overlaid with a kernel specific buffer descriptorstructure. The owning kernel can then use this buffer descriptor to linktogether buffers. In addition (or even as an alternative) the owningkernel can use the owner specific “header” to store kernel specificinformation related to the buffer (for example, a timer associated withthe buffer). FIG. 26 shows an example of how a kernel might use theowner specific “header” 1403. As discussed earlier, a kernel gives upimplicit ownership of a buffer when it passes it to the environment 210or another kernel. Therefore, a kernel should not assume that any fieldsit sets in the owner specific part of the PEP common buffer header 1403will not change in a buffer which it gives away and then gets backunless the semantics of the particular interface are specificallydefined to allow such an assumption. For example, when the BackboneProtocol Kernel 282 passes a PBP segment to the platform environment 210for transmission, the BPK 282 should not assume that any fields it hasdefined in the owner specific “header” have not changed when it gets thebuffer back unless the specific procedural interface definition statesthat this assumption is valid.

[0160] Because a buffer copy is required anyway (because of the way thevarious LAN and WAN “drivers” work), an IP Gateway (i.e., PEP end point210) uses the existing implementation for the LAN Rx buffer 1205, theLAN Tx buffer 1207, the WAN Rx buffer 1209, and WAN Tx buffer 1211. Asingle physical pool of memory is used for both the LAN to WAN and WANto LAN buffer pools 1201 and 1203. The IP Gateway 210 may use a variablesize, single buffer approach for allocating PEP buffers. Single bufferrefers to the fact that only one physical buffer will be required tohold an entire IP packet. Variable size refers to the fact that the sizeof the buffer allocated will exactly match (except for the minimumbuffer size constraint as described above) the size of the IP packet(leaving room for the various buffer headers). The malloc() and free()functions keeps track of the exact size of the buffers. Therefore, theIP Gateway implementation of the PEP end point 210 may not require aplatform specific buffer header.

[0161] With respect to the other PEP end point implementations,Multimedia VSAT buffer handling and Multimedia Relay buffer handling aresimilar to IP Gateway buffer handling. Specifically, the VSATs alsoimplement their LAN to WAN buffer pool 1201 and WAN to LAN buffer pool1203 as pools of memory with buffers allocated using malloc() anddeallocated using free(). A single physical pool of memory is used forboth the LAN to WAN buffer 1201 and WAN to LAN buffer pool 1203. Avariable size, single buffer approach for allocating PEP buffers isemployed. However, unlike the IP Gateway approach, the VSATs include aplatform specific buffer header in each buffer.

[0162] As regards the PES Remote PEP platform environment 210, the useof chains of small buffers is needed to hold IP packets. In order tohide the fact that chained, small buffers are used from the PEP kernels,the PES Remote platform environment 210 needs to ensure that all of theheaders fit into the first buffer of a buffer chain, including the PEPcommon buffer header, not just the protocol headers. To meet thisrequirement, the PES Remote environment does the following for an IPpacket that is received from the local LAN or the WAN. If the length (ofthe content) of the first buffer in the chain is small enough such thatthe PEP common buffer header 1403 can be inserted into the buffer, thecontent of the buffer is shifted to the right to make room for it. Ingeneral, buffer chains are received with all of the buffers full exceptfor the last buffer. Therefore, this condition will, again in general,only be met if the entire IP packet fits into a single small buffer.This option is illustrated in FIG. 27. If the length of the first bufferis too large to allow the PEP common buffer header to be inserted, theenvironment 210 allocates an extra buffer and prepends it to the bufferchain. If no buffer is available, the IP packet is dropped and must berecovered as if it had been dropped crossing the LAN or WAN. The PEPcommon buffer header is placed in the extra buffer and then all of theprotocol headers in the original first buffer are copied into thebuffer. Finally, any data left in the original first buffer is shiftedto the left (to the front of the buffer). This option is illustrated inFIG. 28. While these copies do represent overhead, some sort of copyingis inevitable and this approach should keep the amount of copying to aminimum. In addition, in the PES Remote, the same buffers actually canbe used for the LAN Rx buffer pool 1205, LAN to WAN buffer pool 1201,and (inroute) WAN Tx buffer pool 1211. And, the same buffers 1201, 1205,and 1211 can be used for the (outroute) WAN Rx buffer pool 1209, WAN toLAN buffer pool 1203, and LAN Tx buffer pool 1207. Thus, the PES Remoteplatform environment 210, for example, can avoid some of the copiesrequired in other types of PEP End Point platforms 210 to move data fromone type of buffer to another, offsetting the CPU penalty imposed by thecopies described above. In a PES Remote, the size of a LAN Rx buffer1205, LAN to WAN buffer 1201 and (inroute) WAN Tx buffer 1211 may beeither 146 bytes or 246 bytes (depending upon the particular softwarebuild.). The size of other types of buffers is 246 bytes. Even with thePEP common buffer header 1403, 146 bytes provides ample space for mostnon-data IP packets (e.g., TCP acknowledgements) and for many data IPpackets (e.g., HTTP GETs). In particular, 146 bytes provide issufficient to accommodate any segment or message that needs to begenerated (from scratch) by TSK 280 or BPK 282.

[0163] The platform environment 210 keeps track of the amount of bufferspace being used in each direction for each backbone connection. Thistracking is performed for the purposes of dividing up buffer spaceresources with respect to advertising TCP and PBP windows. At least inthe initial release of the PEP feature, the environment 210 does notbase decisions regarding whether to allocate a buffer on the amount ofbuffer space in use. When the need arises to allocate a buffer, theenvironment 210 allocates the buffer if a buffer is available from theappropriate buffer pool. This policy does not pose any problems in thatTCP and PBP senders (i.e., local TCP hosts and PEP End Point peers) areexpected to not transmit packets beyond what is allowed by theadvertised receive windows they receive from a PEP end point 210. Thispolicy greatly simplifies the error handling associated with theallocation of buffers to send control messages when buffer space isrunning low. The following sections describe tracking and using bufferspace availability to calculate TCP and PBP windows.

[0164] Both TCP and PBP use windows that are sent by the data receiverto the data sender to control how much data can be in transmission fromthe sender to the receiver. In general, a larger window enables higherthroughput. However, throughput is bounded by the size of the smallestlink of the pipe the data is flowing through so, beyond a certain point,an increase in window size no longer increases throughput. To ensurethat the transmitted data is not discarded by the receiver when itarrives, the receiver, in general, bounds the window it advertises basedon the amount of buffer space currently available to receive data.However, in order to use the amount of buffer space available as a boundon window size, the receiver needs to know how much space is available.To support window size calculations based on available buffer space, theplatform environment 210 keeps track of the amount of LAN to WAN and WANto LAN buffer space in use for each backbone connection (in the backboneconnection's environment control block (ECB)). Whenever the environment210 copies a received TCP segment from a LAN Rx buffer 1205 into a LANto WAN buffer 1201, the environment 210 increments the amount of bufferspace in use for the backbone connection which is being used to spoofthe TCP connection to which the TCP segment belongs. The environment 210determines which backbone connection is being used by looking in the CCBof the TCP connection. For the case in which a TCP <SYN> segment isreceived with no CCB allocated for the TCP connection yet, theenvironment 210 counts the buffer of the TCP <SYN> when the TCP SpoofingKernel 280 allocates the CCB. The environment 210 also increments theLAN to WAN buffer space count whenever TSK 280 allocates a LAN to WANbuffer 1201 to generate a TSK message from scratch and whenever theBackbone Protocol Kernel 282 allocates a LAN to WAN buffer 1201 togenerate a PBP segment from scratch.

[0165] WAN to LAN buffer accounting works similar to LAN to WAN bufferaccounting. When the environment 210 copies a received PBP segment froma WAN Rx buffer 1209 into a WAN to LAN buffer 1203, the environment 210increments the amount of buffer space in use for the backbone connectionfrom which the PBP segment was received. In an exemplary embodiment ofthe invention, the environment 210 determines the handle of the backboneconnection by combining the peer index associated with the source IPaddress in the IP packet with the priority of the connection (indicatedby the PBP port number). The environment 210 also increments the WAN toLAN buffer space count when TSK 280 allocates a WAN to LAN buffer 1203to generate a TCP segment from scratch. The environment decrements theLAN to WAN or WAN to LAN buffer space count, as appropriate, whenever abuffer is deallocated. The backbone connection handle used to find theappropriate ECB and a LAN to WAN versus WAN to LAN flag are stored inthe buffer to make deallocation buffer accounting simple. As describedbelow, buffer space is internally tracked in terms of the number ofbuffers in use, not the bytes of buffer space in use.

[0166] In an exemplary embodiment, four types of parameters which areconfigured for a PEP end point 210, affect the use of buffer space todetermine window size advertisement values: (1) per peer buffer space,(2) per peer TCP connection control blocks, (3) per connection resourcespercentage, and (4) maximum window size limit. Referring to FIG. 5, eachPEP end point 501, 503 is configured (via its PEP End Point profile)with the amount of buffer space (specified in units of kilobytes) thatit should use for WAN to LAN traffic received from each of its PEP EndPoint peers. This is the total amount of WAN to LAN buffer space in aremote site PEP End Point 503 (which only has one peer). This is the perpeer WAN to LAN buffer space in a hub site PEP End Point 501. Forexample, if the value configured in the hub site PEP End Point's PEP EndPoint profile is 500 KB, the WAN to LAN buffer pool 1203 for each of itspeers is 500 KB. If there are 100 peers, then total amount of WAN to LANbuffer space is 50 MB. When configuring the WAN to LAN buffer spacevalue, the operator must take into account the total amount of bufferspace available, the number of peers which will need to share the totalpool and the amount of buffer space required in the LAN to WANdirection. The amount of buffer space required in the LAN to WANdirection is nominally the sum of all of the PEP End Point peers' WAN toLAN buffer space values. However, the operator can actually overbookbuffer space; i.e., the operator is not constrained to configure theamount of buffer space to be used such that the total, if all bufferswere in use, is less than the actual amount available. The operatormight do this to cause larger windows to be advertised to improvethroughput (albeit at the risk of dropping packets) or to take advantageof knowledge regarding his applications. For example, the operator mayknow that his applications use more LAN to WAN buffer space during theday and more WAN to LAN buffer space at night. In particular, theoperator will typically overbook the buffer space in the hub site PEPEnd Point 501 because statistically it is very unlikely that trafficwill be being sent to every peer at the same time. Buffer space isspecified in terms of bytes by the operator because the amount of memory(in bytes) is what is known to the operator.

[0167] Internally, for buffer tracking purposes, a PEP End Point 501,503 converts the configured per peer WAN to LAN buffer space value fromthe number of bytes to the number of buffers. This is done by dividingthe number of bytes by the size of a buffer capable of holding a maximumsize IP packet (i.e., 1500 bytes plus the size of the PEP common bufferheader plus the size of any platform specific header). This is performedfor two reasons. First, PBP advertises windows in terms of packets, notbytes, and must assume that every packet it will receive will be amaximum size packet. Second, all buffer space calculations are madeassuming a maximum size IP packet to eliminate assumptions about thebuffer strategy in use in a PEP End Point peer. For example, if a PEPEnd Point uses a variable, exact size buffer strategy and counts bytesbased on actual IP packet sizes but its peer is using a fixed sizebuffer strategy, the byte count will not accurately reflect the amountof memory being used in the peer unless all of the IP packets aremaximum size. Also, the fact that the amount of buffer space can beoverbooked provides a great deal of flexibility with respect to tuningperformance. And, it provides leeway to compensate for assumptions whichmight not apply to a particular customer network. For example, if themaximum size IP packet in a particular network is 1000 bytes instead of1500 bytes and the customer is only using PEP End Point platforms whichuse a variable, exact size buffer strategy, the operator can increasethe WAN to LAN buffer space parameter by 50% to compensate for the useof smaller maximum size IP packets.

[0168] The number of TCP connection control blocks which can be used perPEP End Point peer is also configurable. This value primarily is used todetermine if there is a CCB available in a TSK 280 peer to spoof a newlydetected TCP connection. However, this value also affects buffer spacecalculations related to window sizes because the buffer space availablefor a backbone connection must be divided among all of the TCPconnections which are using the backbone connection. The buffer poolsize calculations are as follows:

S=S _(O)+1500

n _(b) =N _(b) /S,

[0169] where S_(O) is the buffer overhead size in bytes (e.g., PEPCommon Buffer Header and any Platform Specific Buffer Header), S is thebuffer size in bytes, N_(b) is the configured buffer space in bytes, andn_(b) is the buffer space in number of buffers.

[0170] With prioritization, there are potentially multiple backboneconnections between two PEP End Point peers. Therefore, in addition tospecifying the amount of buffer space (and the number of CCBs) for useto a particular PEP End Point peer, the operator needs to specify theallocation of these resources to the various backbone connections. Thisis accomplished via the configuration of resource percentages assignedto each priority of backbone connection on the connectivity profile usedto define the connections. For each priority, the operator assigns aresource percentage ranging from 0% through 100% (with 0% used toindicate that no backbone connection at this priority is required). Theoperator may overbook resources by assigning percentages which add up tomore than 100%. The operator may also underbook resources by assigningpercentages which add up to less than 100%; this might be useful forsetting aside buffer space for use by unspoofed (e.g., UDP) traffic. Theenvironment 210 uses the percentages configured by the operator whenopening backbone connections. The amount of WAN to LAN buffer spaceassigned to a backbone connection is set equal to the per peer WAN toLAN buffer space value multiplied by the resource percentage assigned tothis backbone connection. Similarly, the number of CCBs which can beused with this backbone connection is set equal to the per peer numberof CCBs multiplied by the same resource percentage. In an exemplaryembodiment, different percentage values may be assigned for buffer spaceand CCBs; alternatively, a single parameter may be employed for both.The WAN to LAN buffer space and CCB limit calculations are as follows:

B _(pi) ^(W2L) =n _(b) *X _(pi)

CCB _(pi) =CCB _(e) *X _(pi)

[0171] where X_(pi) is the resource percentage for the backboneconnection to peer “p” at priority “i”, B_(pi) ^(W2L) is the WAN to LANbuffer space limit for the backbone connection to peer “p” at priority“i”, CCB_(pi) represents the CCB limit for the backbone connection topeer “p” at priority “i”, and CCB_(e) is the configured PEP End PointCCB limit. It is noted that the CCB limit is the local limit. The limitthat is used is the smaller of the local CCB limit and the CCB limit ofthe PEP End Point peer.

[0172] While, in general, a TCP or PBP sender can actually deal with aTCP or PBP receiver which shrinks its advertised window (i.e., sends anew window which is smaller than the previous window minus any data sentwithin that window), the protocol operates inefficiently when thisoccurs. Therefore, TCP and PBP receivers are constrained by theirprotocol definitions to not shrink a previously advertised window. Giventhat this is the case, in general, a TCP or PBP receiver should not setits receive window equal to the entire amount of buffer space availablebecause other users of this buffer space may cause the amount of bufferspace to shrink outside of the control of the particular TCP or PBPconnection. In other words, sending a large window reduces flexibilitywith respect to being able to react to reduced buffer availability byslowing down the TCP or PBP sender. Therefore, it is desirable formaximum advertised TCP and PBP window size limits to be enforced. Theselimits represent the largest window a TCP or PBP receiver advertises tothe sender in any segment sent to the sender. It is noted, however, thatif buffer space availability is low, smaller windows (including 0) maybe sent. On the other hand, it is important that, when there is plentyof buffer space, the window that is advertised by a TCP or PBP receiveris large enough to cover the bandwidth * delay product (i.e., the sizeof the pipe divided by the round trip time of the pipe) which applies tothe connection (in order to let the TCP or PBP sender keep theconnection pipe full). Since the round trip time from network to networkcould be different, using hard coded values for the maximum window sizelimits is undesirable. Therefore, these limits may be configured as partof a PEP End Point's PEP End Point profile.

[0173] The PEP End Point profile may include a maximum TCP window sizelimit and a maximum PBP window size limit. Because most TCP connectionsare local to the PEP End Point 210 (connected via Ethernet), a smallmaximum TCP window size may cover the round trip time for most cases.Therefore, in this instance, the maximum TCP window size default may beset to 8 KB. Because of the variety of link speeds that are possible forPBP connections, a default value that works for most situations is notpossible.

[0174] The following discussion describes the calculations that areperformed by the platform environment 210, the TCP Spoofing Kernel 280,and the Backbone Protocol Kernel 282 to convert buffer spaceavailability into advertised receive window sizes. For each backboneconnection, as shown above, the platform environment 210 derives theamount of buffer space that can be used in the WAN to LAN direction forthe connection by multiplying the per peer WAN to LAN buffer space valueby the percentage of the per peer resources which have been allocated tothis backbone connection. The resulting value is then used as the upperbound for WAN to LAN buffer space for this backbone connection. Becausethe per peer WAN to LAN buffer space values may be different in eachpeer, the platform environment 210 cannot directly calculate thecorresponding limit for the amount of LAN to WAN buffer space eventhough the PEP End Point peers may share the same percentage ofresources parameters; instead, this value is provided by the TCPSpoofing Kernel 280. The environment 210 provides the WAN to LAN bufferlimit (and the local number of CCBs limit) to TSK 280 when it opens thebackbone connection. TSK 280 then sends the limit to its TSK peer in aTSK Peer Parameters message. When TSK 280 receives a TPP message, itextracts the peer's WAN to LAN buffer space limit from the message andpasses it to the environment. The environment uses the peer's WAN to LANbuffer space limit as its local LAN to WAN buffer space limit. When abackbone connection is first opened, while waiting for the reception ofa TPP message from the peer, the LAN to WAN buffer space limit and thepeer number of CCBs limit are initialized to 0. This prevents TCPconnections from being spoofed until valid peer parameter information isreceived. As described previously, the platform environment 210 countsthe number of LAN to WAN buffers 1201 and WAN to LAN buffers 1203 it hasallocated to each backbone connection in the backbone connection's ECB.When a buffer is allocated, the appropriate in use count is incremented.When a buffer is deallocated, the backbone connection handle stored bythe environment in the buffer is used to find the proper in use count todecrement. When requested by TSK 280 or BPK 282, the platformenvironment 210 returns the currently available LAN to WAN or WAN to LANbuffer space for a backbone connection. In a platform (e.g. the PESRemote) where small, chained buffers are used, the platform environment210 must normalize its buffer count based on the number of buffersrequired to hold a maximum size IP packet. TSK 280 and BPK 282 use thesevalues to calculate window sizes, as follows:

A _(pi) ^(W2L) =B _(pi) ^(W2L) −U _(pi) ^(W2L)

A _(pi) ^(L2W) =B _(pi) ^(L2W) −U _(pi) ^(L2W),

[0175] where B_(pi) ^(W2L) is the calculated WAN to LAN buffer spacelimit for the backbone connection to peer “p” at priority “i”, B_(pi)^(L2W) is the learned LAN to WAN buffer space limit for the backboneconnection to peer “p” at priority “i”, U_(pi) ^(W2L) is the WAN to LANbuffer space in use for the backbone connection to peer “p” at priority“i”, U_(pi) ^(L2W) is the LAN to WAN buffer space in use for thebackbone connection to peer “p” at priority “i”, A_(pi) ^(W2L) is theWAN to LAN buffer space available for the backbone connection to peer“p” at priority “i”, and A_(pi) ^(L2W) is the LAN to WAN buffer spaceavailable for the backbone connection to peer “p” at priority “i”.

[0176] In addition to the amount of buffer space available, it maybedesirable for a PEP End Point 210 to take into consideration otherfactors when determining window sizes to advertise. In particular, thecurrent latency (generally measured by means of output queue depth) forthe WAN interface 1215 can be an important factor since this interface1215 represents a multiplexing point for the traffic of many competingflows, especially in the hub. In fact, the PEP TCP spoofingimplementation includes the ability to monitor the queue latency andadjust TCP window size advertisements as the queue latency increases anddecreases. When enabled by the operator, the environment 210 may trackqueue latency and use this value to determine a current flow controlfactor. In an exemplary embodiment, the flow control factor may betracked as a percentage from 0% to 100%. When the latency increases bysome operator defined value, the environment 210 may decrease the flowcontrol factor. When the latency decreases by some operator definedvalue, the environment 210 may increase the flow control factor.Nominally, increments of 5% may be used to adjust the flow controlfactor up and down. However, the exact units of increment are notespecially important. Whenever the platform environment 210 receives arequest for the amount of buffer space available in the LAN to WANdirection, it will multiply the result (as determined above) by the flowcontrol factor, as shown below.

A _(pi) ^(L2W) =F*A _(pi) ^(L2W),

[0177] where F is the current flow control factor expressed as apercentage. This results in reduced input from the TCP hosts local tothe PEP end point, when latency increases.

[0178] Utilizing latency to adjust window sizes has applicability to PBPwindows. Notably, queue latency related to sending traffic in the WAN toLAN direction may be employed to adjust the windows that are advertisedby PBP.

[0179]FIG. 29 shows a sliding window used by the PBP, according to oneembodiment of the present invention. Like TCP, PBP uses a sliding window2901 to determine the current acceptable range of sequence numbers. Asshown, the left edge in the sender is the last in-sequence numberacknowledged plus one (Snd_Una). The right edge is equal to the leftedge plus the window size advertised by the receiver. The sender mayfill the window 2901 and upon filling the window 2901 must wait for anacknowledgement in order to transmit new packets. If the window is fulland the sender is given new data to send, it must queue the data forlater transmission after the window 2901 slides. The receiver views thewindow using Rcv_Nxt for the left edge instead of Snd_Una. If a receivedpacket has a sequence number within the window, it is acknowledged. Ifit equals the left edge of the window 2901, a cumulative ACK is usedwhich slides the window 2901 down by one.

[0180] When the TCP Spoofing Kernel (TSK) 280 needs to determine awindow size to advertise in a TCP segment, the TSK 280 starts by callingthe platform environment 210 to get the current LAN to WAN buffer spaceavailability for the backbone connection associated with the spoofed TCPconnection. TSK 280 then divides this number by the number of TCPconnections that are currently using the backbone connection. TSK 280keeps track of the number of TCP connections using a backbone connectionin the backbone connection's TCB, incrementing the count whenever a CCBis allocated and decrementing the count whenever a CCB is deallocated.TSK 280 then converts this value from buffers into bytes by multiplyingthe number of buffers by the MSS being used by the local host to sendTCP segments to TSK 280. This value represents the potential window sizethat can be advertised. However, TSK 280 must make two additional checksbefore using this value. First, the potential value is compared to thewindow size limit. If the potential value is larger than the window sizelimit, the window size limit is advertised instead. If the potentialvalue is smaller than the window size limit, TSK 280 then checks todetermine whether advertising the potential value would shrink thewindow to a value smaller than previously advertised (i.e., would movethe right edge of the rotating window to the left). As indicatedpreviously, a TCP receiver should not shrink its window 2901; therefore,if the potential window value would shrink the window 2901, TSK 280instead advertises the smallest possible window 2901 which does notshrink the previously advertised window (i.e., the value whichrepresents keeping the right edge of the window 2901 in the same place).The calculation of the advertised TCP window 2901 is as follows:

W _(TC) =A _(pi) ^(L2W) /K _(pi) *MSS

W _(TA)=MAX(MIN(W _(TC) ,W _(TL)), W _(TR)),

[0181] where K_(pi) is the current number of TCP connections using thebackbone connection to peer “p” at priority “i”, W_(TC) is thecalculated TCP window 2901, W_(TR) is the TCP window represented by thespace remaining from the previously advertised window (i.e., based onthe last “right edge” advertised), W_(TL) is the configured maximumadvertised TCP window limit, W_(TA) is the TCP window that is actuallyadvertised, and MSS is the TCP connection MSS.

[0182] PBP window calculations are similar to TCP window calculations,except that there may be no need to convert the window to bytes. Whenthe Backbone Protocol Kernel 282 needs to determine a window size toadvertise in a PBP segment, the BPK 282 starts by calling the platformenvironment 210 to get the current WAN to LAN buffer space availabilityfor the backbone connection. This value represents the potential windowsize that can be advertised. However, BPK 282 must make two additionalchecks before using this value. First, the potential value is comparedto the window size limit. If the potential value is larger than thewindow size limit, the window size limit is advertised instead. If thepotential value is smaller than the window size limit, BPK 282 thenchecks to determine whether advertising the potential value would shrinkthe window to a value smaller than previously advertised (i.e., wouldmove the right edge of the rotating window to the left). As statedabove, a PBP receiver should not shrink its window 2901. Therefore, ifthe potential window value would shrink the window 2901, BPK 282 insteadadvertises the smallest possible window 2901 which does not shrink thepreviously advertised window 2901 (i.e., the value which representskeeping the right edge of the window in the same place). The calculationof the advertised PBP window 2901 is as follows.

W _(PC) =A _(pi) ^(W2L)

W _(PA)=MAX(MIN(W _(PC) ,W _(PL)), W _(PR))

[0183] where W_(PC) is the calculated PBP window 2901, W_(PR) is the PBPwindow that is represented by the space remaining from the previouslyadvertised window (i.e., based on the last “right edge” advertised),W_(PL) is the configured maximum advertised PBP window limit, and W_(PA)is PBP window that is actually advertised.

[0184]FIG. 30 illustrates a computer system 3001 upon which anembodiment according to the present invention may be implemented. Such acomputer system 3001 may be configured as a server to execute code thatperforms the PEP functions of the PEP end point 210 as earlierdiscussed. Computer system 3001 includes a bus 3003 or othercommunication mechanism for communicating information, and a processor3005 coupled with bus 3003 for processing the information. Computersystem 3001 also includes a main memory 3007, such as a random accessmemory (RAM) or other dynamic storage device, coupled to bus 3003 forstoring information and instructions to be executed by processor 3005.In addition, main memory 3007 may be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 3005. Notably, PEP controlblocks may be stored in main memory 3007. Computer system 3001 furtherincludes a read only memory (ROM) 3009 or other static storage devicecoupled to bus 3003 for storing static information and instructions forprocessor 3005. A storage device 3011, such as a magnetic disk oroptical disk, is provided and coupled to bus 3003 for storinginformation and instructions.

[0185] Computer system 3001 maybe coupled via bus 3003 to a display3013, such as a cathode ray tube (CRT), for displaying information to acomputer user. An input device 3015, including alphanumeric and otherkeys, is coupled to bus 3003 for communicating information and commandselections to processor 3005. Another type of user input device iscursor control 3017, such as a mouse, a trackball, or cursor directionkeys for communicating direction information and command selections toprocessor 3005 and for controlling cursor movement on display 3013.

[0186] Embodiments are related to the use of computer system 3001 toperform the PEP functions of the PEP end point 210. According to oneembodiment, this automatic update approach is provided by computersystem 3001 in response to processor 3005 executing one or moresequences of one or more instructions contained in main memory 3007.Such instructions may be read into main memory 3007 from anothercomputer-readable medium, such as storage device 3011. Execution of thesequences of instructions contained in main memory 3007 causes processor3005 to perform the process steps described herein. One or moreprocessors in a multi-processing arrangement may also be employed toexecute the sequences of instructions contained in main memory 3007. Inalternative embodiments, hard-wired circuitry may be used in place of orin combination with software instructions. Thus, embodiments are notlimited to any specific combination of hardware circuitry and software.

[0187] The term “computer-readable medium” as used herein refers to anymedium that participates in providing instructions to processor 3005 forexecution the PEP functions of the PEP end point 210. Such a medium maytake many forms, including but not limited to, non-volatile media,volatile media, and transmission media. Non-volatile media includes, forexample, optical or magnetic disks, such as storage device 3011.Volatile media includes dynamic memory, such as main memory 3007.Transmission media includes coaxial cables, copper wire and fiberoptics, including the wires that comprise bus 3003. Transmission mediacan also take the form of acoustic or light waves, such as thosegenerated during radio wave and infrared data communications.

[0188] Common forms of computer-readable media include, for example, afloppy disk, a flexible disk, hard disk, magnetic tape, or any othermagnetic medium, a CD-ROM, any other optical medium, punch cards, papertape, any other physical medium with patterns of holes, a RAM, a PROM,and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrierwave as described hereinafter, or any other medium from which a computercan read.

[0189] Various forms of computer readable media may be involved incarrying one or more sequences of one or more instructions to processor3005 for execution. For example, the instructions may initially becarried on a magnetic disk of a remote computer. The remote computer canload the instructions relating to execution of the PEP functions of thePEP end point 210 into its dynamic memory and send the instructions overa telephone line using a modem. A modem local to computer system 3001can receive the data on the telephone line and use an infraredtransmitter to convert the data to an infrared signal. An infrareddetector coupled to bus 3003 can receive the data carried in theinfrared signal and place the data on bus 3003. Bus 3003 carries thedata to main memory 3007, from which processor 3005 retrieves andexecutes the instructions. The instructions received by main memory 3007may optionally be stored on storage device 3011 either before or afterexecution by processor 3005.

[0190] Computer system 3001 also includes one or more communicationinterfaces 3019 coupled to bus 3003. Communication interfaces 3019provide a two-way data communication coupling to network links 3021 and3022 which are connected to a local area network (LAN) 3023 and a widearea network (WAN) 3024, respectively. The WAN 3024, according to oneembodiment of the present invention, may be a satellite network.Communication interface 3019 may be a network interface card to attachto any packet switched LAN. As another example, communication interface3019 may be an asymmetrical digital subscriber line (ADSL) card, anintegrated services digital network (ISDN) card, a cable modem, or amodem to provide a data communication connection to a corresponding typeof telephone line. Wireless links may also be implemented. In any suchimplementation, communication interface 3019 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

[0191] Network link 3021 typically provides data communication throughone or more networks to other data devices. For example, network link3021 may provide a connection through local area network 3023 to a hostcomputer 3025 or to data equipment operated by an Internet ServiceProvider (ISP) 3027. ISP 3027 in turn provides data communicationservices through the Internet 505. In addition, LAN 3023 is linked to anintranet 3029. The intranet 3029, LAN 3023 and Internet 505 all useelectrical, electromagnetic or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 3021 and through communication interface 3019which carrythe digital data to and from computer system 3001, are exemplary formsof carrier waves transporting the information.

[0192] Computer system 3001 can send messages and receive data,including program code, through the network(s), network link 3021 andcommunication interface 3019. In the Internet example, a server 3031might transmit a requested code for an application program throughInternet 505, ISP 3027, LAN 3023 and communication interface 3019. Thereceived code may be executed by processor 3005 as it is received,and/or stored in storage device 3011, or other non-volatile storage forlater execution. In this manner, computer system 3001 may obtainapplication code in the form of a carrier wave. Computer system 3001 cantransmit notifications and receive data, including program code, throughthe network(s), network link 3021 and communication interface 3019.

[0193] The techniques described herein provide several advantages overprior approaches to improving network performance, particularly in apacket switched network such as the Internet. A local PEP end point anda remote PEP end point communicate to optimize the exchange of datathrough a backbone connection through the use of performance enhancingfunctions. This approach advantageously minimizes network latency.

[0194] Obviously, numerous modifications and variations of the presentinvention are possible in light of the above teachings. It is thereforeto be understood that within the scope of the appended claims, theinvention may be practiced otherwise than as specifically describedherein.

What is claimed is:
 1. A network apparatus for providing performanceenhancements of a communication network, comprising: a plurality ofcommunication interfaces configured to receive and to forward messagesaccording to a prescribed protocol; a plurality of modules configured toprocess the messages to effect performance enhancing functions; and aplurality of buffers configured to store the received messages andmessages that are generated by one of the plurality of modules, whereina portion of the plurality of buffers is shared by the plurality ofmodules based upon execution of a particular one of the performanceenhancing functions, each of the plurality of buffers has a datastructure that includes an expandable header to accommodate differentmessage types.
 2. The network apparatus according to claim 1, whereinthe plurality of modules comprises a spoofing module configured toperform selective spoofing, a connection module configured to multiplexa plurality of connections over a common backbone connection, aprioritization module configured to prioritize access to the backboneconnection, and a path selection module configured to determine a pathamong a plurality of paths to transmit the received messages.
 3. Thenetwork apparatus according to claim 1, wherein the communicationinterface includes a local area network (LAN) interface, and a wide areanetwork (WAN) interface, one of the plurality of buffers beingdesignated as a LAN-to-WAN buffer that stores the receive messages in aLAN-to-WAN direction, another one of the plurality of buffers beingdesignated as a WAN-to-LAN buffer that stores the receive messages in aWAN-to-LAN direction.
 4. The network apparatus according to claim 3,wherein the WAN is satellite network.
 5. The network apparatus accordingto claim 1, wherein the data structure of the plurality of bufferscomprises: a specific header field that stores platform specificinformation; a common header field the stores information known to theplurality of modules; a payload field; an offset field that indicatesstart of the payload field; and a header growth field that provides avariable header length.
 6. The network apparatus according to claim 5,wherein the common header field comprises: a flag field that specifiesdirection of message flow; a connection handle field that specifieshandle of a backbone connection; and an owner specific field that storesan owner specific header.
 7. The network apparatus according to claim 1,wherein the prescribed protocol is the Transmission Control Protocol(TCP).
 8. A method for providing performance enhancements of acommunication network, the method comprising: receiving messagesaccording to a prescribed protocol; processing the messages to effectperformance enhancing functions via a plurality of modules; and storingthe received messages and messages that are generated by one of theplurality of modules in a plurality of buffers, wherein a portion of theplurality of buffers is shared by the plurality of modules based uponexecution of a particular one of the performance enhancing functions,each of the plurality of buffers has a data structure that includes anexpandable header to accommodate different message types.
 9. The methodaccording to claim 8, wherein the plurality of modules in the storingstep comprises a spoofing module configured to perform selectivespoofing, a connection module configured to multiplex a plurality ofconnections over a common backbone connection, a prioritization moduleconfigured to prioritize access to the backbone connection, and a pathselection module configured to determine a path among a plurality ofpaths to transmit the received messages.
 10. The method according toclaim 8, wherein the receiving step is performed by a communicationinterface that includes at least one of a local area network (LAN)interface and a wide area network (WAN) interface, one of the pluralityof buffers being designated as a LAN-to-WAN buffer that stores thereceive messages in a LAN-to-WAN direction, another one of the pluralityof buffers being designated as a WAN-to-LAN buffer that stores thereceive messages in a WAN-to-LAN direction.
 11. The method according toclaim 10, wherein the WAN is satellite network.
 12. The method accordingto claim 8, wherein the data structure of the plurality of bufferscomprises: a specific header field that stores platform specificinformation; a common header field the stores information known to theplurality of modules; a payload field; an offset field that indicatesstart of the payload field; and a header growth field that provides avariable header length.
 13. The method according to claim 12, whereinthe common header field comprises: a flag field that specifies directionof message flow; a connection handle field that specifies handle of abackbone connection; and an owner specific field that stores an ownerspecific header.
 14. The method according to claim 8, wherein theprescribed protocol in the receiving step is the Transmission ControlProtocol (TCP).
 15. A network apparatus for providing performanceenhancements of a communication network, comprising: means for receivingmessages according to a prescribed protocol; and means for processingthe messages to effect performance enhancing functions, wherein thereceived messages and messages that are generated by processing meansare stored in a plurality of buffers, a portion of the plurality ofbuffers being shared by the processing means based upon execution of aparticular one of the performance enhancing functions, each of theplurality of buffers having a data structure that includes an expandableheader to accommodate different message types.
 16. The network apparatusaccording to claim 15, wherein the processing means comprises a spoofingmodule configured to perform selective spoofing, a connection moduleconfigured to multiplex a plurality of connections over a commonbackbone connection, a prioritization module configured to prioritizeaccess to the backbone connection, and a path selection moduleconfigured to determine a path among a plurality of paths to transmitthe received messages.
 17. The network apparatus according to claim 15,wherein the receiving means includes at least one of a local areanetwork (LAN) interface and a wide area network (WAN) interface, one ofthe plurality of buffers being designated as a LAN-to-WAN buffer thatstores the receive messages in a LAN-to-WAN direction, another one ofthe plurality of buffers being designated as a WAN-to-LAN buffer thatstores the receive messages in a WAN-to-LAN direction.
 18. The networkapparatus according to claim 17, wherein the WAN is satellite network.19. The network apparatus according to claim 15, wherein the datastructure of the plurality of buffers comprises: a specific header fieldthat stores platform specific information; a common header field thestores information known to the plurality of modules; a payload field;an offset field that indicates start of the payload field; and a headergrowth field that provides a variable header length.
 20. The networkapparatus according to claim 19, wherein the common header fieldcomprises: a flag field that specifies direction of message flow; aconnection handle field that specifies handle of a backbone connection;and an owner specific field that stores an owner specific header. 21.The network apparatus according to claim 15, wherein the prescribedprotocol is the Transmission Control Protocol (TCP).
 22. Acomputer-readable medium carrying one or more sequences of one or moreinstructions for providing performance enhancements of a communicationnetwork, the one or more sequences of one or more instructions includinginstructions which, when executed by one or more processors, cause theone or more processors to perform the steps of: receiving messagesaccording to a prescribed protocol; processing the messages to effectperformance enhancing functions via a plurality of modules; and storingthe received messages and messages that are generated by one of theplurality of modules in a plurality of buffers, wherein a portion of theplurality of buffers is shared by the plurality of modules based uponexecution of a particular one of the performance enhancing functions,each of the plurality of buffers has a data structure that includes anexpandable header to accommodate different message types.
 23. Thecomputer-readable medium according to claim 22, wherein the plurality ofmodules in the storing step comprises a spoofing module configured toperform selective spoofing, a connection module configured to multiplexa plurality of connections over a common backbone connection, aprioritization module configured to prioritize access to the backboneconnection, and a path selection module configured to determine a pathamong a plurality of paths to transmit the received messages.
 24. Thecomputer-readable medium according to claim 22, wherein the receivingstep is performed by a communication interface that includes at leastone of a local area network (LAN) interface and a wide area network(WAN) interface, one of the plurality of buffers being designated as aLAN-to-WAN buffer that stores the receive messages in a LAN-to-WANdirection, another one of the plurality of buffers being designated as aWAN-to-LAN buffer that stores the receive messages in a WAN-to-LANdirection.
 25. The computer-readable medium according to claim 24,wherein the WAN is satellite network.
 26. The computer-readable mediumaccording to claim 22, wherein the data structure of the plurality ofbuffers comprises: a specific header field that stores platform specificinformation; a common header field the stores information known to theplurality of modules; a payload field; an offset field that indicatesstart of the payload field; and a header growth field that provides avariable header length.
 27. The computer-readable medium according toclaim 26, wherein the common header field comprises: a flag field thatspecifies direction of message flow; a connection handle field thatspecifies handle of a backbone connection; and an owner specific fieldthat stores an owner specific header.
 28. The computer-readable mediumaccording to claim 22, wherein the prescribed protocol in the receivingstep is the Transmission Control Protocol (TCP).
 29. A memory forstoring information for providing performance enhancements of acommunication network, the memory comprising a data structure including:a specific header field that stores platform specific information; acommon header field the stores information known to the plurality ofmodules; a payload field; an offset field that indicates start of thepayload field; and a header growth field that provides a variable headerlength.
 30. The memory according to claim 29, wherein the common headerfield comprises: a flag field that specifies direction of message flow;a connection handle field that specifies handle of a backboneconnection; and an owner specific field that stores an owner specificheader.